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Dear Rural, Small School Leader: 

The need for alternative assessment methods is evolving alongside a growing shift from 
schools that are content-centered to schools that are learner-centered. Higher order 
thinking processes, decision making skills and cooperative grouping strategies are just a 
few of the myriad student proficiencies that schools are choosing to assess. The range of 
articles in this Rural, Small Schools Network Information Packet includes a discussion of 
assessment standards, an overview of a variety of assessment methods, and descriptions 
of 'state of the art' science and math programs. 

Students are assessed on what they learn; as well as how they respond to a test. In the 
past educators have relied heavily on one testing instrument. Multi-faceted assessment 
instruments like portfolios, essays, multiple choice questions, and hands-on presentation 
instruments now provide a more in-depth perspective of what students have learned as 
well as offering students a variety of ways to demonstrate their knowledge. 

We hope you find this packet informative as you look at assessment as a way of 
gathering information about students' learning processes. An evaluation card has been 
provided so you can send us your feedback. We also welcome your suggestions for 
future Information Exchange Packet topics. Please jot any ideas you may have on the 
card or contact us at the Rural, Small Schools Network, 83 Boston Post Road, Sudbury, 
MA 01776, (508) 443-7991. 



Sincerely, 





John R. Sullivan, Jr., Ed.D. 

Program Director 

Rural, Small Schools Network 



Virginia L. Warn 
Associate Program Director 
Rural, Small Schools Network 



Sfr\insi\f\\ EnaLiml. \c\\ )(irk. hit^rto Rin). iiini thf L'.S. Virdn IsUmds 
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Standards, Assessment, and 
Educational Quality 

by 

Lauren B. Resnick 



National performance 
standards can play a 



American schools are 
underachieving institutions. 
Their aspirations are too low and 
they are not working up to 
capacity. The reason lies not in 
students learning less than before. 
Indeed, scores on tests of "basic skilb" have risen over the 
past two decades.' But students are not learning the skilb 
and knowledge they will need in the future. Why is this 
so? 

Schooling as we know it today was designed early in 
this century.* It aimed to educate a small elite of future 
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vital role in systemic 
education reform. 



leaders — managers, engineers, 
physicians, lawyers, and other 
professionals — to use their minds 
well. For the majority of students, 
however, educational aims were 
much more modest. The goal was 
to teach basic citizenship and to inculcate the limited 
skilb young people would need to nke their places as 
workers in an economy needing many more willing hands 
than active minds. It was a mass production form of 
education suited to the mass production economy of the 
time.' Broad, liberating education for the many was 
considered unnecessary and unachievable. American 
tesQ, the standardized tests chat repoR on children's and 
schoob' "grade level", are the products of mass production 
education.* They assessstudenQ* command ofdisembodied 
bits of information, not their ability to analyze complex 
situations or marshal knowledge to solve problems.' Yet 
these tests substantially control what is uught in schoob. 

The strategy seemed to work for many decades: 
Living standards rose for more arid more Americans and 
American democracy seemed secure.* But the world at 
the beginning of the next century will bear few rewards for 
iiulividuab or natioiu that limit themselves to educatmg 
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only a few to chink.' To maintain a high'wage economy, 
almost all individualswtll have to think their way through 
their workdays — analyzingprcblems, proposing solutions, 
troubleshoocingai¥iiepaiiingequipmenc,communicatitig 
with others, and managing resources of time arul materials. 
For the first time since the Industrial Revolution, the 
human resource needs of a vibrant economy and the civic 
requirements of a truly participatory democracy are 
converging. The time has come ibr American schools to 
set their sights higgler, to move from their inherited 
preoccupation with low>level fact and skill learning to 
the goab of thinking, reasoning, and problem'Solvit\g for 
every student. 

Doing so will require the most thorough revision of 
aspitanon and practice that any set of irvttitutiotts has 
ever knowt\. It will necessitate settiitg new standards of 
quality and mobilizing all resources to ensure that every 

The time has come for 
American schools to set their 
sights higher and address the 
goals of thinking, reasoning 
and problem-solving. 

single child, regardless of the child's race, language, 
origin, or presumed native ability, has the learning 
opportunities needed to meet those standards. It will 
mean re>educating teachers to work in new ways and 
giving them the authority they will need to set a new 
course with their students. New methods of assessing 
student achievement will be essential in this 
transformation. 

New forms of assessment, assessments based on 
complex task performances, scored by trained and 
thoughtful judges, can release educators from the grip of 
testing programs that drive irutructional attention away 
from thoughtfulness and complex applications of 
knowledge. They can exemplify new standards for 
achievement and provide clear representations of what 
students should now be striving to learn and teachers 
teach. 

Standards and assessment alone, however, will work 
no magic. Real educational improvement requires 
interlocking and coherent changes in several components 
of the education system. These include curriculum, 
textbooks, teacher preparation, and continuing 
professional development. Further, for these elements to 
sustain themselves, there will need to be fundamental 



changes in the way schools are managed and in how 
schoob relate to families, communities, and the social 
service delivery system. Stimulating and establing these 
interlc-ckingchanges coruritutes as^rstenucrcform policy — 
the only kind of policy likely to produce the new leveb of 
student achievement that are sought. 

PERFORMANCE STANDARDS AND 
SYSTEMIC CHANGE 

Implementing systemic change requires attention 
not only to how the elements of a system function 
together in equilibrium, but to how they might influence 
one another in a period of disequilibrium, and which 
elements might be most susceptible to organized. ; 
intennonal modification. In some countries, an effon at . 
national improvement of education would begin with ' 
attention to curriculum and, perhaps, organizational : 
structure, including teacher certification requirements.** 
Both of these would fall under the purview of a ministry 
of education with the power, afrer consultation appropriate 
to the country's processes, to impose the curriculum and 
the new forms of organization. Textbooks, exams, and 
the content of teacher education programs would change 
in due course in response to the mandated curriculum. 

In countries that, like us, have traditions of local 
rather than natiot\al conaol of education, this centralized 
approach to changing the system is not available. Some 
countries, most notably Britain, have responded to this 
condition by overthrowing significant aspects of the 
local control tradition and moving toward a national 
curriculum.* We in the United Statesare crying to induce 
systemic changes without federalizing education and 
without creating a controlling national curriculum. We 
need a different point of departure. What are the 
possibilities? 

Three frequently proposedstartingpoints for reform 
are textbooks, teacher education, and education 
governance. All are crucial to a full program of systemic 
reform. But none of the three serves as a promising 
starting point. Consider textbooks first. These serve as a 
kirul of it facto natioiuil curriculum, and changes in 
textbooks would strot\gly influence whac is taught and 
learned. But because textbook publishers respond 
primarily to market incentives, a radical change in demaivi 
from purchasers (i.e., educators) must be created. 
Textbook publishirig can be expected to follow, not lead, 
the reform effort. 

The prospeco are not much better for teacher 
education as a staning point. The argumencfor reforming 
teacher education is compelling; only a quite differently 
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prepared teaching force will be able to educate $tuder\ts 
ir\ the new, more demanding ways that are required for 
the future. But teacher education in this country is 
largely controlled by institutions with even greater 
traditions of "local" control than the public schoob. 
Individual faculty at colleges and universities substantially 
control their own programs of instruction. No one has 
proposed a conviiKii^; and, at least for the moment. 



We in the United States are 
trying to induce systemic 
cliange witl-iout federalizing 
education and creating a 
controlling national 
curriculum. 



politically viable means of directly inducing the hundreds 
of institutions responsible for educating teachers to make 
radical changes in what they do themselves or demand of 
hicure teachers. Furthermore, even with major changes 
in pre-service teacher education, it would take a long 
time to change schooling practice. Newcomers to any 
profession are not well positioned to take the lead in 
changing practice. For many years to come, the vast 
majority of teachers in place will have been educated in 
the "old" ways. Means must be found to more directly 
affect the practices of those already in the teaching force. 

What about changing the management structure of 
education — giving local educators and parents more direct 
decision-making power? There seems little doubt, for 
reasons discussed below, that such changes are essential 
to overall systemic reform. Apart from a few schoob with 
substantial outside resources, however, changes in 
management and decision making have, so fiar, rarely 
produced significant changes in curriculum, teaching, 
and learning. Those that have reached such results have 
almost always depended upon exceptioruil individual 
leaders. '° These leaders often negotiate special privileges 
for the school — for example, freedom from certain 
mandated tests during an experimental period, the right 
to choose faculty in ways that are not standard in their 
district — and sometimes raise extra funds from 
fbundatiotu and other donors. Such schoob are tolerated 
as exceptional experimems that do not really challenge 
the normal ways in which the system worb. Without a 
change in the surrounding system, exceptional schoob of 
thb kind are fragile and likely to be driven back to 
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ordinary ways of proceeding as soon as they become too ! 
vbibly succes^l or their special advocates move on lo ' 
new challenges. ^j^k 
Performance standarcb are a promuing start for^^ 
systemic change. We need an education system in which 
good schoob that have high expectations for students 
and that work hard to meet them can thrive — where 
good schoob are the norm rather than exceptions needing 
special protection. We should organize education so that 
it does not depend upon a few exceptiotul people putting 
out extraordinary effort, but relies instead upon thousands 
of competent and committed people working in concert 
with, rather than against, the "system." Perforinancc 
standards represent the best means to achieve this. Such 
standards provide tangible goab that students can strive 
to achieve. Standards also allow teachers to measure 
student performance against an objective criterion of 
excellence. If we can agree on luitional standards for 
stud>!nt achievement and create conditions in school 
systems all over the country in which those statxlards are 
internalized and made the centerpiece of educators' and 
students' efforts, there is a good probability that 
curriculum, professional development, textbooks, and, 
eventually, teacher preparation can be changed so that 
the entire system is working toward the standards. 



INTERNALIZED STANDARDS: KEY TO A 

NEW CULTURE OF TEACHING 

j 

Thb approach may sound like a prescription for j 
federal, or at least natiorul, tests or exams tied to standards ; 
set by a committee of experts. It is not. National standards ! 
alone will not achieve our goab. Standards must be < 
mumalized. Unless standards are held as personal goals — | 
by teachers first, and eventually by students — little in the i 
way of profound educational change can result from a i 
standard-setting and assessment process. Let us consider j 
why thb is so. 

If a national test embodying the highest standards 
of a thinking curriculum could be created in Washington 
and given to every student in America, with test scores 
sent back to schoob from a national office, educators 
might hope to change their school's performance — 
especially if some coruequences for themselves or their 
students were attached to test scores. But if educators did 
not understand in a profound manner the differences 
between high and low scores, thsy would not know how 
to direct their efforts. Additionatl .', if educators were not 
peisonally invested in the standards and assessments 
based on them, the various forms of cheating aiul"working 
around the system" that have been amply documenQ^i in 



55 



LAUREN B. RESNICK 



studies of test-based, hig^-stakes reform eiifons would be 
likely to domiiute educate: respotue." 

For these reardns, an eiifective'role for the federal 
government — or any national agency — in educational 
reform cannot be the apparently simple one of imposing 
standards from above or afar. The real task is not to 
announce new standards, but to create an educational 
culture in which working to achieve those standards 
pervades the system because the standards have become 
the internalized goals of everyone in the system 

Leading businesses working to become high- 
performance organizations understand this point. They 
do not anempt to raise quality by puning in tougher 
reject standards at the end of the production line. Instead, 
they mount a complex process of engaging employees to 
understarul new business requirements, reorganize their 
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Unless standards are held as 
personal goals by teachers 
and students, little in the way 
of profound educational 
change can result. 
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I work patterns, and set the quality standards toward 
which they will work. A new culture, not a set of 
mechanical procedures, is at the hean of a transformation 

i in quality. 

Education can do no less. Changing standards and 
I assessments by themselves will not substantially change 
{ real learning. Those changes will come only when 
i teachers act as professionals who internalize and 
{ voluntarily work toward the standards of their field. 
I Promoting that internalization must lie at the core of any 
'; strategy for educational reform. 

DEVELOPING INTERNALIZED 
STANDARDS: SOME PRINCIPLES 
FOR ACTION 

I How can we achieve this goal? We need some 
principles for designing astandards-based systemic reform 
effort. The principles that follow guide the work of the 
New Standards Project — a partnership of seventeen 
states and six major school districts which have joined 
together to develop alternative approaches to sening 



education standards and assessing student achievement. 
They are working to develop a system of assessments in 
which varied local assessments can be benchmarked to a 
shared set of national standards. 

Engage Teachers as Central Actors in StaxuLard 
Setting and Assessment Development 

This approach is not limited to a few teachers wh :> 
represent the teacher point of view in a committee of ! 
experts, but rather every teacher whose students will be ' 
judged by the standards and assessment. Teachers in ' 
schools throughout the country must be involved in 
developing assessment tasks and scoring student work. 
The primary actors in the work of the New Standards j 
Project have been teachers. Teachers have been our 
initial assessment task developers. Nearly 500 teachers i 
panicipated in administering performance assessments 
in the New Standards fourth grade trial assessments in ' 
mathematics and literacy last spring. This same group, 
led by a set of specially trained lead teachers, will also be 
responsible for scoring the assessments.'^ 

! Create a Culture Incorporating Professional 
Discussion of Standards for Student Work 

This guideline represents an extension of the , 
previous principle, specifying houi the engagement of ! 
teache» in the standards process ought to proceed. 
Imagine that all the teachers involved in scoring j 
assessments participated in extensive professional ' 
discussion of why particular student performances ■ 
warranted particular scores. Teachers working in this '. 
kind of a standards and assessment system would become 
self-conscious, reflective judges of student work. Further, 
if they panicipated from time to time not just in applying ! 
scoring criteria developed by others but in developing ; 
criteria and arguing for them, the standards would come | 
to be "owned" by the teachers. Such standards would be : 
internalized. Teachers would be motivated not just to j 
raise scores, but to meet quality standards they understood, i 
believed in, and had some degree of control over. In the 
New Standards Project, teachers administering the new 
assessments panicipated in orienution discussions in 
which curriculum was discussed and responses of students 
to the assessment tasks were reviewed. As the project 
unfolds, new panicipating teachers w ill become members 
of ongoing work groups in which they generate and | 
critique tasks, analyze student performances, and design | 
and apply scoririg criteria. 
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Embed Most Assessment m Regular Cturiculum 
Work Rather Thatt m "One Shot" Examinations 

Assessmena need not be limited to what students 
can do in scheduled, supervised exams. Experiments in 
many individual schools, and now in two entire states 
(Vermont and Kentucky) show that collected portfolios 
of students' work, much like the portfolios anists prepare 
for art juries, can provide rich evaluations of students' 
capabilities.'* These portfolios might contain the results 
of extended projects, exhibitiorts prepared by the student.- 
and special performances, thus applying to the academic 
realm the practices of the visual and performing arts, 
sports, and scouting "merit badges." In a portfolio system, 
assessments are not separate (torn the curriculum; they 
are pan of it, a special window on the regular work of the 
student. The New Standards assessment system will 
place portfolios of individual student work at its heart. 
Scheduled performance exams will be used primarily to 
anchor and audit ponfolio'based scores. 

There are many reastins to use such curriculum* 
embedded assessments as the major methods to judge 
student performance. Not least, the extended activities 
that can be included in portfolios come closer to reflecting 
deep educatioiial goab than do scheduled «xam questions. 
Such assessments also give students multiple and varied 
opportunities to show their competencies. But one 
fundamental benefit is that curriculum'embedded 
assessments, by definition, can permeate the teaching 
and learning process. In the most effective uses of 
portfolio assessment, students, working with their 
teachers, play a major role in selecting what should be 
presented as their representative and htst work. Perhaps 
the most important benefit of such a portfolio process 
involves its capacity to develop self-reflective judgment 
capac ities on the part of both students and teachers. This 
is the kind of internalization process for which we are 
aiming. 

Make Standard Setting a Public Process 

Teachers and other educators must be central actors 
in developing education standards, but they caimot be 
solely resporuible. Education is a social good, in which 
many elements of society have a stake. Unless there is a 
general consertsus about educational goals and standards, 
educators wilt be subject to continual pressures that will 
weaken ordisable their efforts to meetstandards they may 
have adopted. In many European countries, after exams 
are given, examination questions are published and 
discussed on television and radio shows. We need an 



American version of this kind of public participation, 
one that can engage alV segment* of our varied 
communities — parents, employers, community and child 
advocates — in active discussion of education goals and 
standards. In die New Standards Project, we will be 
developing ways to use print and broadcast media to 
create opportunities for citizen groups to discuss examples 



Without a general consensus 
about educational goals 
and standards, educators 
will be subject to pressures 
that will weal<en or disable 
their efforts. 



of students' work on performance exams and curriculum- 
embedded assessments. Focus groups we have already run 
in several locations around the country show that people : 
of many diffierent backgrounds and levels of education are 
able to engage productively in such discussions.'* Our 
task now is to work with states and localities to make ^jjjk 
opportunities for such discussions broadly available and 
to develop procedures for systematically using their 
opinions as part of a formal standard'Setting process. 

APPROPRIATE GOVERNMENT ROLES 



Given this view of the role of standards and 
assessments in educational reform, what roles are 
appropriate for federal, state, and local agencies? Clearly, 
building central national tests intended for administration 
to all children in the nation would not be productive. 
Internalized standards, not externally imposed ones, are 
the only likely meatu of creating broad changes in 
educational effotr Even making administration of a 
national test voluntary would not help much. It is 
unlikely that individual teachers, schoob or communities 
would be given an opportunity to volunteer, and they 
would certainly have little chance to directly influence 
assessment content or process. Thus, the effect of a 
voluntary, but centrally developed, single national test 
would not be very different from an imposed one. 
Assessments, like the Specifics of curricula to which they 
are tied, need to engage.the energies of those closest to 
instruction. Thus, assessments will need to include 
substantial local variation. 
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At die same time, broadly specified national 
curriculum Standards, sueh as those that now exist for 
mathematics, can play a vital role in systemic education 
reform. In state after sate today, and in many school 
districo as well, educator and citizen groups are grappling 
with the problem of setting goab and standards for their 
education systems. The need for well thoughtniut 
frameworks for speciftc reform is clear In the New 
Standards Project, for example, development of 
mathematics assessments has proceeded more smoothly 
than literacy assessments, primarily because the standards 
created by the National Council of Teachers of 
Mathematics (NCTM) provided a common framework 
from the start.'^ At the same time, the NCTM standards • 
provide plenty of room for localities — whether states or 
irxdtvidual districts — to craft curricula and assessments 
suitable for their own communities. As an example, one 
NCTM standard used in grades K-4 for the mastery of j 



It is States and localities — not 
tt-ie federal government— 
that will have the primary 
responsibility for putting 
programs of systemic reform 
into place. 



geometry is the ability to describe, model, draw, and 
classify shapes. " Few have the resources to engage in the 
kind of standards development -vork that NCTM has 
done at a natioaal level. Thus, an important role for the 
federal government involves suppon of national content 
standards development for other parrs of the curriculum. 
Fortunately, the U.S. Department of Education has begun 
to fund development of national content standards, and 
we can now look forward to having such standards for 
most of the major school subjects. 

It is states and localities, not the federal government, 
that will in the end have the primary responsibility for 
putting programs of systemic ceform, including standards 
and assessments, into place. Traditionally, education has 
largely been a power reserved to thr^tates." Efforts by 
the federal government to directly impose curricula could 
be subject to legal challenge. But to say that the power to 
impose standards resides exclusively at the state level is to 
ignore the substantial American tradition of much more 
local control of eHv^ation. Furthermore, state'tmposed 
assessments wotiMnvot bf automatically more conducive 



to intemaliza:ion of standards by educators and 
communities than national ones. To convince ourselves 
of this, we need only recall that it is state, not national, 
tests that have been used in the mandated assessment 
programs in which test scores have been allegedly 
manipulated." To reap the benefits of internalized 
standards, states will need to leave much discretion in the 
matter of assessment to local education authorities and 
even iiuiividual schoob. A portfolio system, in which 
some assessment tasks are common to large numbers of 
students and others are locally selected by schoob and 
individual teachers, can provide the necessary balance 
between local tnitiarive and statewide standards. 

Localities want curricula, and thus assessme:tts, 
crafted to their own needs. At the same time, they want 
to know that they are competitive beyond the confines of 
their local communities; few want to educate their children 
in ways that would not permit them to prosper beyond 
local borders. To achieve these objectives, methods must 
be found to Itnic the varied assessments that will be 
developed by localities to shared state and national 
standards. Simple statistical formulas for converting 
scores on one test into equivalent scores on another are 
unlikely to work; to legitimately apply such formulas, the 
assessments would need to besoalike that it isquestionable 
that local options would be practicable.-' A number of 
proposab for alternative ways of linking assessments are 
under discussion. These include administenng a national 
"reference exam" to a sample of students in a state or 
dbtrict, including common natiotul "anchor tasks'* in a 
state or district's assessments, and arranging for "cross- 
grading" of one jurisdiction's exams by another's trained 
scorers.-^ None of these, however, is appropriate if 
different assessments are not first established as 
"reasonable" tests of the same content and objectives for 
learning. This can only be determined through an 
organized process of social and professional judgment. 
For this reason, a standards certification board of some 
kind will almost certainly be needed as part of a national 
assessment system.^ 

CONCLUSION 

The program outlined here is both ambitious and 
optimistic. It assumes that in the future American 
students will need to learn skilb in new ways and at much 
higher standards than ever before. This approach suggests, 
in tuYn, that teachers will need to teach in new ways and 
with different expectations for their students than many 
have held in the past. Performance standards, represented 
in assessments, are a powerful means to begin the process 
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of systemic educational change. Such standards provide 
tangible goab that students can strive to achieve, allowing 
teachers to measure student performance against an 
1 objective criterion of excellence. Assessments can be 
I tailored to local school environments yet liiUced with 
! broader state and national standards. Extensive 
j participation by teachers in designing and implementii^ 
new assessments can serve as pan of a professional 
development program that will transform the new 
standards into internalized, persoiuil goals of teachers 
and students throughout the country. Standards and- 
; assessments used in this way promise to prepare America 
to corApete with brain rather than brawn in the next 
'■ century.^ 

I NOTES 

i 

! ' Linda Darling-Hammomi, The Implicaacns of Ttsdng Policy for 
Quality and Equality, 73 Phi DaTA Kappan 220, 221 (1991 ). 

•Ray MAiLSHAa& Mari.Tucker, Thinking Rm a Livind: Eixk jvixx 

^NPTHE WE.^LTHi1FNATK1N.>i 1J.27 (1992). 



'Id. 



"C/. Darling-'Haminond, supra note 1, at 221. 

" Manhatl & Tucker, supra note 2, at 1 10. 

' 'Memben of the New Standards Project are: the states of Atixona. 
Arkansas, California. Coiocado, Connecticut, Delaware, Florida. 
Iowa, Kentucky. Maine. New York. Oregon, South Carolina. 
Texas, Vetniont. Virginia, and Washington; and the distncts of 
Forth Worth. PittsKirgh, Rochester. San Diego, White Plains, and 
New York City. Together they enroll ctoie to 50% ofall U.S. public 
school chiklten. New Standards is a joint undertaking of the 
Learning Research and Development Center at the Univeisity of 
Pitobuigh and the NatiotulCenteron Education and the Economy 
in Rochester, New York. The is described in Lauren B. 

Resnkx & Marc Tucker, National Cnt on Educ and the 
Economy. The New Standarijs Procct 1992-1995. A Profosal 
(1992). 

'♦M.at40. 

Darling-Hammond, supra note I, at 224. 

'* Vincent J. BRrajo, Natkxai. Ctr. on Eouc and the Economt . 
PuBuc Response to the New Standards Program ( 1991 ). 

1 

1 

! " Natkjnai. Qxincil of Teachers of Mathematics, CuRRia.XLM 

; AND EVaLUATKTN STANDARDS FOR SCHlXX MATHEMATICS ( 1989). 

j 

i '"M.at48. 



' DarlinR-Hammund, supra note 1, at 221-222. 
^ Manhall & Tucker, supra note 2, at 13-27. 
*ld. at 31. 
•Id. at 64-69. 

.'Commission on THESKiusof the American Worktorce. National 
Ctr. on Eouc. and th,; Economy, America's Choice: High Skius 
OR Low Wages! 94-102 (1990). 

" Paul Maston. Responst on Poor Pupils Was 'Too Slow' . The Daily 
Telegraph, Dec. 16. 1991, at 2. 

'"Manhall & Tucker, supra note 2, at 109-1 10. 



I ■* Charles E Faber, Is Local Control of die Sdiools Soil a VuHt 
Opoon.', 14 Harv. J.L & Pub. Pol. 447. 

Darling-Hammond, supra note 1, at 221. 

Robert L Linn, Educatiotul Assessment: Expanded Expectatkin* 
atKlChallenges, Addressat the American Psychological Assocuciun 
Annual Meeting (Aug. 16, 1992). 

"Robert L Lnn, National Ctr. for Research on Evaluat**.. 
Standards and Student Testtnc, Cross-State CoMPARABiLm , » 
JuocmentsofStuoent WRrriNC: Results from the New Stand*» t • 
Proiect, 21-22 (1991). 

" National counql on educ. standards and testing, raimv 

standards for AMERICAN EDUCATION 35-36 (1992). 



14 



I 



O . WINTER 1992-93 59 

BEST COPY AVAILABLE 



Reprinted with permission from PHI DELTA KAPPAN . May 1989. 

A True Test: 

Toward More Authentic and 
Equitable Assessment 



WHEN AN educauonal 
problem pemstt de- 
spite the well-inten- 
tioned effons of many 
people to solve it, it's 
a safe bet that the problem hasn't been 
properly framed. Assessment in educa- 
tion has clearly become such a problem, 
since every sute reports above-average 
scores on norm-referenced achievement 
tests and since everyone agrees (paradox- 
ically) that such tests shouldn't drive in- 
struction but that their number and in- 
fluence should nevertheless increase.' 
More ominously, we seem unable to see 
any moral harm in bypassing context- 
sensitive human judgments of human 
abilities in the name of sutistical accura- 
cy and economy . 

We haven't moved beyond lamenting 
these problems, because we have failed 
to stop and ask some essential questions: 
Just what are tests meant to do? Whose 
purposes do they (and should they) serve? 
Are large-scale testing programs neces- 
sary? When are tests that are designed 
to monitor accountability harmful to the 
educational process? Need they be so in- 
trusive? Is there an approach to uphold- 
ing and examining a school's standards 
that might actually aid learning? 

But we wont ^ fw in answering these 
questions until we ask the most basic one: 
What is a true test? I propose a radical 
answer, in the sense of a remm to the 
roots; we have lost sight of the fan that 
a true test of intellectual ability requires 
the performance of exemplary tasks. First, 
authentic assessments replicate the chal- 
lenges and standards of performance that 
typically face writers, businesqwople. sci- 

GRANT WIGGINS is a senior associate with 
the Nanonai Center on Education and the 
Economy, Rochester , NX, and a special con- 
sultant on assessment for the Coalizion of Es- 
sential Schools. 



As long as we hold simplistic monitoring tests to be 
adequate models of and incentives for reaching national 
intellectual standards, Mr. Wiggins warns, studem 
performance, teaching, and our thinking and discussion 
about assessment will remain flaccid and uninspired 
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eadstt, eomnamity leaden, dengnen, or 
historians. These include writing essays 
and reports, conducting individual and 
group research, designing proposals and 
nxxk-ups, assembli^ poitfolios, and so 
on. Second, legitimate assessments are 
responsive to individual students and to 
school contextt. Evaluation is most ac- 
curate and equitable when it entails bu> 
man judgment and dialogue, so that the 
person tested can ask for clarification of 
questions and explain his or b»r answers. 

A genuine test of intellectual achieve- 
mem doesn't merely check "standardized" 
work in a mechanical way. It reveals 
achi^emett on the essentials, even if they 
are not easily quantified. In other words, 
an authentic test not only reveals student 
achievement to the examiner, but also re- 
veals to the test-taker the actual chal- 
lenges and standards of the field. 
. To use a medical metaphor, our con- 
fusion over the uses of standardized tests 
is akin to mistaking pulse rate for the to- 
tal effect of a healthful regimen. Stan- 
dardized tests have no more effect on a 
smdent's intellectual health than taking a 
pulse has on a patient's physical health. 
If we want standardized tests to be au- 
thentic, to help snidents learn about them- 
selves and about the subjea tnaner or field 
being tested, they must become more than 
merdy indicators of one superficial symp- 
tom. 

Reform begins, then, by recognizing 
that the test is central to instruction. Any 
tests and final exams inevitably cast their 
shadows on all prior work. Thus they not 
only monitor standards, but also set them. 



Students acknowledge this truth with Ifaeir 
plaintive query. Is this going to be on the 
test? And their instinos are conect; we 
should not feel despair about such a view. 
The teiK always sea th& de £Kto standards 
of a school despite whatever else is 
proclaimed. A school sAiotii<f leach to the 
test." The catch is that the test must of- 
fer students a genuine intellectual chal- 
lenge, and teachers must be involved in 
designing the test if it is to be an effec- 
tive point of leverage. 



SCmNG STANDARDS 

We need to recognize from the out- 
set that the problems we face are more 
ecological (i.e., political, structural, and 
economic) than technical. For example, 
Norman Frederiksen, a senior research- 
er with the Educational Testing Service 
(ETS), notes that "simational tests are not 
widely used in testing programs because 
of considerations having to do with cost 
and efficiency."^ In order to overcome 
the resistance to using such simational 
tests, we must make a powerftil case to 
the public (and to teachers habiniated to 
shon-answer tests as an adequate meas- 
ure of ability) that a standardized test 
of intellecmai ability is a contradiction 
in terms. We must show that influential 
"monitoring" tests are so ineievant (and 
even harmful) to genuine intellectual stan- 
dards that their cost - to student learn- 
ing and teacher professionalism - is too 
high, however financially efficient they 
may be as a means of gathering dau. 

The inescapable dilemma presented by 
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sing authentic 
Standards and 
tasks to judge 

intellectual 
ability is labor' 
intensive and 
time'Consuming. 



mass testing is that using auhentic stan- 
dards and tasks to judge intellectual abil- 
ity is labor-inteiuive and time-consum- 
ing. Examinen must be trained, and mui- 
tiple, contexoial tests of the snidents must 
be conducted. Genuine testt also make it 
more difficult to compare, rank, and sort 
because they rarely involve one simple, 
definitive test with an unambiguous re- 
suit and a single residue number. There- 
fore, as long as tests are thought of only^^ 
in temu of accounubility, read reforms^B 
will be thwarted. After all, why do we^i^ 
need to devise more expensive tests if 
current dau are reliable? When we fac- 
tor iu the self-interest of test companies 
and of colleges and school districts, we 
can see that resistance to reform is like- 
ly to be strong. 

The psychoroetricians and the account- 
ants are not the villains, however. As I 
have noted elsewhere, teachers fail to im- 
derstand their own unwitting role in the 
growdt of standardized tesdng.* Mass 
assessment resulted from legitimate con- 
cern about the failure of the schools to 
set clear, justifiable, and consistent stan- 
dards to whiizh it would bold its gradu- 
ates and teachers accounuble. But the 
problem is still with us: high school tran- 
scripts tell us nothing about what a sm- 
dent can actually do. Grades and Car- 
negie units hide vast differences between 
courses and schools. An A in 1 Ith-grade 
English may mean merely that a sudent 
was dutiful and able to fUI in blanks on 
worksheets about juvenile novels. And it 
renuuns possible for a student to pass all 
of his. or her courses and still remain 
functionally and culturally illiterate, jflk 

But the solution of imposing an effi^lP 
ciem and "standard* test has an uglier his- 
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tory. The tests grew out of the 'school- the educationtl enterprise, such reduc- 

efficiency" movement in the years be- tionist shoncuts, such high student/teach- 

tween 191 1 and 1916, a time oddly simi- er ratios, and such dys^nctionai ailoca- 

lar to our own. The movement, spear- tion of time and resources will be seen 

headed by the work of Franklin Bobbitt, as intolerable . 
was driven by crude and harmful analo- Schools and teachers do nor tolerate the 

gies drawn from Frederick Taylor's man- same kind of thinking in athletics, uSe 

agement principles, which were used to aru, and clubs. The requirements of the 

improve faaory production. Raymond game, recital, play, debate, or science 

Callahan notts that the reformers, then fair are clear, and those requirements de- 

as now, were fir too anxious to atisiy termine the use of time, the assignment 

external critics and to reduce complex in- of personnel, and the allocation of mon- 

teliecnml standards and teacber behaviors ey. Far nK)re time - often one's spare 

to simple numbers and traitt.* Implicit- time - is devoted to insuring adequate 

ly, there were signs of hereditarian and practice and success. Even in the poorest 

social-class-based views of intelligence; schools, the ratio of playen to inter- 

the tests were used as sotting mechanisms scholastic coaches is about 12 to l.> The 

at least partly in response to the increased test demands such dedication of time; 

heterogeneity of the school population as coaching requires one-to-one interaction, 

a resuh of the influx of immigrants.^ And no one complains about teaching to 

The "standards" were usually cast in the test in athletic competition. sure thorough analysis and useful feed- 
terms of the increased amount of work We need to begin anew, from the back to students about results, 
to be demanded of teachers and snidents. premise that a testing program must ad- This reversal in thinking will make us 
As George Strayer, head of the National dress questions about the ineviubie im- pay more attention to what we mean by 
Education Association (NEA) Commit- pact of tests (and scoring methods) on evidence of knowing. Mastery is more 
tee on Tests and Standards for School Ef- students and their learning. We must ask than producing verbal answers on cue; 
ficiency, reported, '^e may not hope to different questions. What kinds of chal- it involves thpughtAil understanding, 
achieve progress except as such measur- lenges would be of most educational val- as well. And thoughtful tuiderstanding 
ing sticks are available." A school su- ue to students? What kinds of challenges implies being able to do something ef- 
perintendem put it more bluntly: Tne would give teachers useful information fective, transformative, or novel with a 
results of a few well-planned tests would about the abilities of their students? How problem or complex situation. An au- 
carry more weight with the businessman will the resulu of a test help students thentic test enables us to watch a learner 
and parent than all the psychology in the know their strengths and weaknesses on pose, uckle, and solve slightly ambigu- 
world."* essential tasks? How can a school ade- ous problems. It allows us to watch a 

Even with unionization and the insights quateiy communicate its standards to in- student marshal evidence, arrange argu- 
gained from better education, modem terested outsiden and justify them, so ments, and tak? purposeful action to ad- 
teachers still hil prey to the insistent that standardized tests become less neces- dress the problems.* Understanding is 
claims of noneducation interests. The sary and less influential? often best seen in the ability to criticize 
wishes of college admissions officers, of or extend knowledge, to explain and ex- 
employers, of budget makers, of sched- plore tiie limits and assumptions on which 

ulers, and even of the secretaries who AUTHENTIC TESTS ^ theory restt. Knowledge is thtis dis- 

enter grades on computers often take Tests should be central experiences in played as thoughtfiil know-how - a 

precedence over the needs of snidents to learning. The problems of adminiso^- blend of good judgment, sound habits, 

be properly examined and the needs of tion, scoring, and between-school com- responsiveness to the problem at hand, 

teachers to deliberate and confer about parisons should come only after an au- and control over the appropriate informa- 

effective test design and grading. thentic test had been devised - a rever- tion and context. Indeed, genuine mas- 

Thus, when teachers regard tests as sal of the current practice of test design, tery tisually involves even more: doing 
something to be done as quickly as pos- If we wish to design an authentic test, something with grace and style, 
sible after leaching" has ended in order we must first decide what are the actual To prove that an answer was not an ac- 
to shake out a final grade, they succumb performances that we want studenu to be cident or a thoughtless (if correct) re- 
to the same flawed togic employed by the good at. We must design those perform- sponse, multiple and varied tests are re- 
test companies (with far less sutistical ances first and worry about a fair and quired. In performance-based areas we 
justification). Such acquiescence is pos- thorough method of grading them later, do not assess competence on the basis of 
sible only when the essential ideas and Do we judge our students to be deficient one performance. We repeatedly assess 
priorities in education are unclear or have in writing, speaking, listening, artistic a student's work — through a portfolio 
been lost. If tests serve only as adminis- creation, finding and citing evidence, and or a season of games. Over time and in 
trative monitors, then short-answer, *ob- problem solving? Then let the tests ask the context of numerous performances, 
jective" tests - an ironic misnomer^ - them to write, speak, hsten, create, do we observe the patterns of success and 
will suffice (particularly if one teaches original research, and solve problems, failure and the reasons behind them. 
128 students and has only a single day Only then need we worry about scoring Traditional tests - as arbitrarily timed, 
in which to grade final exams). However, the performances,, trt ining the judges, superficial exercises (more like drills on 
if a test is seen as the heart and soul of and adapting the. school calendar to in- the practice field than like a game) that 



To design an 
authentic test, we 
must first decide 
what are the actual 
performances that 
we want students 
to be good at. 
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are given only once or twice - leave us 
with no way of gauging a snident's abili- 
ty to make progress over time. 

We typically learn too much about a 
student's shottHeim recall and too little 
about what is most important: a student's 
habits of mind. In talldng about habiu of 
mind, I mean something more substan- 
tive than "process* sidlls divorced from 
context - the formalism decried by E. 
D. Hirsch and others. For example, a 
new concept — say, irony or the formu- 
la F « ma — can be learned as a habit 
or disposition of mind for effortlessly 
handling information that bad previous- 
ly been confusing.)* As the word habit 
implies, if we are serious about having 
stuidenu display thoughtful control over 
ideas, a single performance is inadequate. 
We need to observe students' repertoires, 
not rote catechisms coughed up in re- 
sponse to pat questions. 

The problem is more serious than it 
first appean. The difficulty of learning 
lies in the breaking of natural but dys- 
functional habits, libe often-strange qtul- 
ity of new knowledge can cause us to un- 
wittingly misunderstand new ideas by as- 
similating them into our old conceptions; 
this is particularly true when instruction 
is only verbal. That is why so many sm- 
dents who do well on school tests seem 
so thoughtless and incompetent in solv- 
ing real-world problems. For example, 
the research done at Johns Hopkins Uni- 
versity demonstrates how precarious and 
illusory "Tcnowledge" of physics really is, 
when even well-trained students habini- 
ally invoke erroneous but plausible ideas 
about force on certain problems.*' 

The true test is so central to instruc- 
tion that it is known from the stan and 
repeatedly taken because it is both cen- 
tral and complex - equivalent to the 
game to be played or the musical piece 
to be perfonmed. The true test of abili- 
ty is to perform consistently well tasks 
whose criteria for success are known 
and valued. By contrast, questions on 
standardized tests are usually kept "se- 
cure." hidden from students and teachers, 
and they thus contradict the most basic 
conditions required for learning. » (Of 
course, sutistical validity and reliability 
depend on the test being secret, and, 
when a test is kept secret, the questions 
can be used again.) 

Designing authentic tests should in- 
volve knowledge use that is forward- 
looking. We need to view tests as 'assess- 
ments of enablement," to bonow Robert 
Glaser's term. Rather than merely judg- 



Most so-called 
"criterion-referenced" 
tests are inadequate 
because the 
problems are 
contrived, and the 
cues are artificial. 



ing whether students have learned what 
was uught, we should "assess knowledge 
in terms of its constructive use for Av- 
ther learning. . . . [We should assess 
reading ability] in a way that takes into 
account that the purpose of learning to 
read is to enable [students) to learn from 
reading."'* All tests should involve sni- 
dents in the actual challenges, standards, 
and habiu needed for success in the aca- 
demic disciplines or in die workplace: 
conducting original research, analyzing 
the research of others in thr service of 
one's research, arguing critically, and 
synthesizing divergem viewpoints. With- 
in reasonable and reachable limits, a real 
test replicates the authentic intellecmal 
challenges facing a person in the field. 
(Such tesu are usually also the most en- 
gaging.) 

The practical problems of test design 
can best be overcome by thinking of aca- 
demic tests as the intellectual equivalent 
of public "performances." To enable a 
student is to help him or her make prog- 
ress in hartdling complex tasks. Tht nov- 
ice athlete and the novice actor face the 
same challenges as the seasoned profes- 
sional. But schod tests make the complex 
simple by dividing it into isolated and 
simplistic chores — asif tbesudentneed 
not practice the true test of performance, 
the test of putong all the elements togeth- 
er. This apparently logical approach of 
breaking tasks down into their compo- 
nents leads to tesu that tstess only ar- 
tificially isolated "outcomes" and provide 
no hope of stimulating genuine intellec- 
mal progress. As a result, teaching to 
such tesu becomes mechanical, sutic, 
and disengaging. Coaches of musicians, 
actors, debaters, and athletes know bet- 



ter. Thty know that what one learns in 
drill is never adequate to produce mas- ( 
tery. ^ 
That is why most so<aIled "criterion- 
referenced" tesu are inadequate: the 
problems are contrived, and the cues are 
artificial. Such tesu remove what is cen- 
tral to inteUecnial competence: the use of 
judgment to recognize and pose complex 
problems as a prelude to using one's dis- 
crete knowledge to solve them. Authen- 
tic challenges - be they essays, original 
research, or artistic performances - are 
inherently ambiguous and open*ended. 
As Frederiksen has said: 

Most of the inpoitant problems one 
^Kcs ire tU-ttncnired, as are all the 
really important social, political, and 
scientific problems. ... But Ul-stiue- 
nired problems are not found in stan- 
dardized achievement tesu. . . . Effi- 
cieat tesu tend to drive oat leu effi- 
cient tests, leaving many important 
abilities uneited and uniaugbL . . . All 
this reveals a problem when we con- 
sider the influence of an accountabili- 
ty system in education. ... We need 
a much broader conception of what a 
test is." 

Put simply, what the snident needs is 
a test with more sophisticated criteria for 
judging performance. In a truly authen- 
tic and criterion-referenced education, far 
more time would be spent teaching and 
testing the student's ability to understand 
and internalize the criteria of genuine 
competence. What is so harmful about 
current teaching and testing is that they 
frequently reinforce - unwittingly - the 
lesson that mere right aiuwers, put forth 
by going through the motions, are ade- 
quate signs of ability. Again, this is a 
mistake rarely made by coaches, who 
know that their hardest and most impor- 
tant job is to raise the standards and ex- 
pectations of their smdenu. 



EXAMPLES OF AtnHENTIC TESTS 

Let US examine sooK tesu and criteria 
devised by teachen woridng to honor the 
ideas Fve been discussing uader the head- 
ing of "exhibition of mastery" — one of 
the nine "Common Principles" around 
which members of the Coalition of Es- 
sential Schools have organized their re- 
form effortt.i* Here are two examples 
of final exams that seem to iq>licate more 
accurately the challenges facing experu 
in the fidd. 

An oral history projea for ninth*grad- 
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en. ** You must complete an onl history 
based on interviews and wricien sources 
and present your findings orally in class. 
The choice of subject matter will be up 
to you. Some exaiiq>les of possible topics 
include: your family, running a small 
business, substance abuse, a labor union, 
teenage parents, or recent immigrants. 
You are to create three workable hypoth- 
eses based on your preliminary investi- 
gations and come up with four questions 
you will ask to test each hypothesis. 

To meet the criteria for evaluating the 
oral history project described above, you 
must: 

• investigate three hypotheses; 

• describe at least one change over 
time; 

• demonstrate &at you have done back- 
ground research; 

• interview four appropriate people as 
sC'Urces; 

• prepare at least four questions relat- 
ed to each hypothesis; 

• ask questions that are not leading or 
biased; 

• ask follow-up questions when ap- 
propriate; 

• note important differences between 
fact and opinion in answers that you re- 
ceive; 

• use evidence to suppon your choice 
of ^e best hypothesis; and 

• organize ycur writing and your class 
presentation. 

A course-ending simulation/exam in 
economics." You are the chief execu- 
tive officer of an established fam. Your 
frnn has always capmred a major share 
of the market, because of good use of 
technology, understanding ot the natural 
laws of constraint, understanding of mar- 
k^x systems, and the maintenance of a 
high standard for your product. How- 
ever, in recent months your product has 
become pan of a new trend in public 
tastes. Several new firms have entered 
the market and have captured pan of your 
sales. Your product's proportional share 
of total aggregate demand is continuing 
to fall. When demand returns to normal, 
you will be controlling less of the mar- 
ket than before. 

Your board of direaors has given you 
less than a month to prepare a repon that 
solves the problem in the shon run and 
in the long run. In preparing the repon, 
you should: 1) defme the problem, 2) 
prepare dau to illustrate the current sit- 
uation, 3) prepare dau to illustrate con- 
ditions one year in the fumre. 4) recom- 
mend acdon for today, 5) recommend ac- 



tion over the next year, and 6) discuss 
where your company will be in the mar- 
ket six months fh>m today and one year 
from today. 

The tasks that must be completed in the 
course of this project include: 

• deriving formulas for supply, de- 
mand, elasticity, and equUibrium: 

• preparing schedules for supply, de- 
mand, costs, and revenues; 

• graphing all work; 

• preparing a written evaluation of the 
current and ^ture situation for the mar- 
ket in general and for your company iu 
panictilar; 

• preparing a written recommendation 
for your board of directon; 

• showing aggregate demand today 
and predicting what it will be one year 
hence: and 

• showing the demand for your firm's 
product today and predicting what it will 
be one year hence. 

Connecticut has implemented a range 
of performance-based assessments in sci- 
ence, foreign languages, drafting, and 
small-engine repair, using experts in the 
field to help develop apt performance 
criteria and test protocols. Here is an 
excerpt from the Connecticut manual 
describing the performance criteria for 
foreign languages; these criteria have 
been derived firom the guidelines of the 
American Council on the T eaching of 
Foreign Languages (ACTFL)." On the 
written test, smdents are asked to draft 
a letter to a pen pal. The four levels used 
for scoring are novice, intermediate, in- 
termediate high, and advanced; they are 
differentiated as follows: 



• Soviet. Swdenu use high-finequen- 
cy words, memorized {ritrases, and for- 
mulaic sentences on familiar topics. Sm- 
dents show little or no creativity with the 
language beyond the memorized paoems. 

• Iniemediau. Students recombine the 
learned vocabulary and sttucmres into 
simple sentences. Sentences are choppy, 
with frequent erroR in grammar, vocabu- 
lary, and spelling. Sentences will be very 
simple at the low end of the intermedi- 
ate range and will often read very much 
like a direct translation of Engl^h. 

• Inxermediaxe high. Student can write 
creative sentences, sometimes fturly com- 
plex ones, but n^". consistently. Structural 
forms reflecting time, tense, or tspta. are 
attempted, but the resuh is not always 
successful. Student show an emerging 
ability to describe an*, narrate in para- 
graphs, but papers often read like aca- 
demic exercises. 

• /4^vonce<f. Smdents are able to join 
sentences in simple discourse and have 
sufficient writing vocabulary to express 
themselves simply, although the language 
may not be idiomatic. Studentt show 
good control of the most frequently used 
synuctic structures and a sense that they 
are comforuble with the target language 
and can go beyond the academic task. 

Of course, using such an approach is 
time-consuming, but it is not impractical 
or inapplicable to all subject areas on 
a large scale. The MAP (Monitoring 
Achievement in Pittsburgh) testing pro- 
gram offers testt of cntical thinking and 
writing that rely on essay questions and 
are specifically designed to provide di- 
agnostic information to teachers and 
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students. Pittsburgh is also working, 
through its Sylltbus-Driven Exam Pro- 
gnun, to devise exemplary test items that 
are based more closely on the curricu- 
lum. »♦ 

On die state level, Vermont has recent- 
ly announced that it will move toward a 
portfolio-based assessment in writing and 
mathematics, drawing 00 the woric of the 
various afliliates of the National Writing 
Projea and of the Assessment of Per- 
formance Unit (APU) in Great Britain. 
California has piloted perfbnsiance-based 
tesu in science and other subjects to go 
with iu sutewide essay-vmtiog test. 

RESPONSIVENESS AND EQUHY 

Daniel Resnick and Lauren Resnick 
have proposed a different way of mak- 
ing many of these points. They have 
argued that American studenu are the 
'most tested" but the "least examined" 
youngsters in the world.* As their epi- 
gram suggests, we rarely honor the origi- 
nal meaning of the word test. Originally 
a testum was a porous cup for determin- 
ing the purity of metal; later it came to 
stand for any procedures for detetmin- 
ing the worth of a person's effort. To 
prove the value or ascertain the nature of 
a student's understanding implies that ap- 
pearances can deceive. A coirea answer 
can disguise thoughtless recall. A student 
might quickly correct an error or a slip 
that obscures thoughtful understand- 
ing; indeed, when a snident's reasoning 
is heard, an error might not actually be 
an error at all. 

The root of the word assessment re- 
minds us that an assessor should *sit with" 
a learner in some sense to be sure that 
the student's answer really means what 
it seems to mean. Does a correa answer 
mask thoughtless recall? Does a wrong 
answer obscure thoughtful understand- 
ing? We can know for sure by asking fur- 
ther questions, by seeking explanation or 
substantiation, by requesting a self-as- 
sessment, or by soliciting the student's re- 
sponse to the assessment. 

The problem can be cast in broader 
moral terms: the standardized test is dis- 
respeaful by design. Mass testing as we 
know it treats students as objects — as 
if their education and thought processes 
were similar and as if the reasons for 
their answers were irrelevant. Test-takers 
are not, therefore, treated as human sub- 
jects whose feedback is essential to the 
accuracy of the assessment. Pilot stan- 
dardized tests catch many technical de- 
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es to higher-order questions are inherent- 
ly unprediccUile. 

The standardized test is thus inherent- 
ly inequittble. I am using the word eq- 
uity in iu original, philosophical mean- 
ing, as it is incorporated into the British 
and American legal systems. The concept 
is commonsensical but profound: blank 
laws and policies (or standardized tests) 
are inherently unable to encompass the 
ineviuble idiosyncradc cases for which 
we ought always to make excepdons to 
the rule. ArisuxleputttbestiTheequiu- 
ble is a correction of the law where it is 
defective owing to its universality ."2> 

In the context of testing, equity re- 
quires us to insure that human judgment 
is not overrrm or made obsolete by an ef- 
ficient, mechanical scoring system. Ex- 
ternally designed aixi externally mandat- 
ed tests are dangerously immune to the 
possibility that a student might legid- 
mately need to have a question rephrued 
or might deserve the opportunity to de- 
fend an unexpected or "incorrect" an- 
swer, even when the test questions are 
well-structured and the answers are mul- 
tiple choice. How tnany times do teach- 
ers, parents,. or employers have to alter 
an evaluation after having an answer or 
action explained? Sometimes, studenu 
need only a hint or a slight rephrasing to 
recall and use what they know. We rely 
on human judges in law and in athletics 
because complex judgmenu cannot be 
reduced to niles if they are to be truly 
equiuble. To gauge understanding, we 
must explore a student's answer; there 
must be some possibility of dialogue be- 
tween the assessor and the assessed to in- 
sure that the student is fuUy examined. 

This concern for equity and dialogue 
is not idle, romantic, or esoteric. Con- 
sider the following example from the Na- 
tional Assessment of Educational Prog- 
ress (NAEP) science test. Learning by 
Doing, which was piloted a few years 
ago.c On one of the tasks, snidents were 
given three sets of sutistics that sup- 



posedly derived from a mini-Olympics 
that some children had suged (see Table 
1). The introductory text noted that die 
children "decided to make each event of 
the same importance." No other infonsa- 
tion that bears on the question was pro- 
vided. The test presented the snidents 
with the resulu of three events from the 
competition. 

The first question asked. Who would 
be the all-around winner? The scoring 
manual gives diese instructions: 

Score 4 poinu fbr accurate ranking 
of the children's perfonnaoce on each 
event and citing Zabi as the overall 
winner. Score 3 points for using a rank- 
ing approach ... but misinterpreting 
perfcxinanoe on the dash event . . . and 
therefore, dting the wrong winner. 
Score 2 poinu for a response which 
cites an overall winner or a tie with 
an explanation that demonstrates some 
recognition that a quamittove means 
of comparison is needed. Sooie 1 point 
if the snident nukes a selection of an 
overall winner with an irrelevant or 
non-quantitstive account or without 
providing an explanation. Score 0 for 
no response. 

Makes sense, right? But now ask your- 
self how, using the given criteria, you 
would score the following response giv- 
en by a third-grader: 

A. Who would be the all-around 
winner? 

No one. 

B. Explain how you decided who 
would be the all-around winner. Be 
sure to show your woric. 

No one is' the all-around winner. 

The NAEP scorer gave the answer a 
score of 1 . Given the criteria, we can see 
why. The student failed to give an ex- 
planation or any numerical odculadons 
to suppon the answer. 

But could that answer somehow be 
apt in the mind of the student? Could 
it be that the 9-year-old deliberately 
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tod correctly answered "no ooe," since 
'all-around* covld mean 'Nnnner of aU 
events"? If looked at in this way, couldn't 
it be that the child was more thoughtful 
than most by deliberately not taking the 
bait of pan B (which presumably would 
have caused the child to pause and con- 
sider his or her answer), llie fiill sen- 
tence answer in pan B - remember, this 
is a 9-year-old - is revealing to me. It 
is more emphatic than the answer to pan 
A, as if to say, "Your question suggests 
I should have found one all-around win- 
ner, but I won't be fooled. I stick to my 
answer that no one was the all-around 
winner." (Note, by the way, that in the 
scorer's manual the word all-cround has 
been changed to overall.) The student did 
not, of course, explain the answer, but 
it is conceivable tluit the instruction was 
confusing, given that there was no Vork" 
needed to determine that *no one' was the 
all-around winner. One quick follow-up 
question could have settled the matter. 

A moral question with inteUectual ram- 
ifications is at issue here: Who is re- 
sponsible for insuring that an answer has 
been fully explored or understood, the 
tester or Uie smdent? One reason to safe- 
guard the teacher's role as primary asses- 
sor is that the most accurate and equita- 
ble evaluation depends on relationships 
that have developed over time between 
examiner and smdent. The teacher is the 
only one who knows what the student can 
or cannot do consistently, and the teach- 
er can always follow up on confusing, 
glib, or ambiguous answers. 

In this country we have been so en- 
amored of efficient testing that we have 



overlooked feasible in-class alternatives 
ro such impersonal testing, which are 
already in use around the world. The 
German abitur (containing essay and 
oral questions) is designed mi sowed by 
classroom teachen, who submit two pos- 
sible tests to a sute board for approval. 
The APU in Great Britain has for more 
than a decade developed tests that are 
designed for classroom use and that in- 
volve interaction between assessor and 
student. 

What is so striking about many of the 
APU test protocols is that the assessor is 
meant to probe, prompt, and even teach, 
if necessary, to be sure of the smdent's 
actual ability and to enable the learner to 
learn from the assessment. Is many of 
these tests the first answer (or lack of 
one) is not deemed a sufficient insight 
into the smdent's knowledge.^ Consid- 
er, for example, the following sections 
from the assessor's manual for a mathe- 
matics test for British IS-year-olds cover- 
ing the ideas of perimeter, area, and cir- 
cumference. 

1. Ask: "What is the perimeter of a 
recungle?* [Write student answer.] 

2. Present sheet with rectangle 
ABCD. Ask: "Could you show me the 
perimeter of this rectangle?" // ntees- 
sary, teach. 

3. Ask: "How would you measure 
the perimeter of the rectangle?" If nte- 
essary, prompt for full procedure. If 
necessary, teach. . . . 

10. "Estimate the leafih of the cir- 
cumference of this circle." 

11. Ask: "What would you do to 
check your estimate?" (String is on 
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the ttble.] If no response, prompt for 
string. 

13. Ask: "Is there any other meth- 
od?" If student does not suggest using 
C - nd. prompt with, "Would it help 
to measure the diameter of the cixde?" 

The scoring system works as follows: 
1) unaided success; 2) success following 
one prompt from the tester; 3) success 
following a series of prompts; 4) teach- 
ing by the tester, prompts unsuccessful; 
S) an unsuccessftil response, and tester 
did not prompt or teach; 6) an unsuc- 
cessful response despite prompting and 
teaching: 7) question not given; and 8) 
unaided success where student correa- 
ed an unsuccessful attempt without help. 
The "successfiil" responses were com- 
bined into two larger categories called 
"unaided success" and "aided success," 
with percenuges given for each.>* 

The Australians for years have used 
similar tasks and similariy trained teach- 
ers to conduct distria- and sutewide 
assessnMnts in a c ade m ic subject areas 
(much as we do in this country with the 
Advanced Placement exams). Teachers 
give tests made up of questioiu drawn 
from banks of agreed-upon items and 
then mark tbem. Reliability is achieved 
through a process called "moderatioo," in 
which teachers of the same subjecu gath- 
er to compare resulu and to set criteria 
for grading. 

To instire that professionalization is 
aided, not undermined, by natioiud test- 
ing, the process of "group moderation' 
has been made a central feature of the 
proposed new national aises s m en t system 
in Great Britain. The tesu will be both 
teacher-given and s t andaidtoed. But what 
is so admirable — ud equitable — is that 
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the process of group moderation requires 
collective judgments about any discrep- 
ancies between grade patterns in differ- 
ent schools and between results in a giv- 
en school and on the nationally stan- 
dardized criterion-referenced test. Sig- 
nificantly, the process of moderation can, 
on occasion, override the results of the 
nationally standardized test: 

A Hrst tuk of a modention group 
would be to examine how weU the pat- 
tenu of the two matched fbr each group 
of pupils [comparing percenuges of 
students assigned to each level]. . . . 
The meedng could then go on to ex- 
plore discrepancies in the paneni of 
particular schools or groups, using 
samples of pupils' work and burwUdge 
of the circumsumces of schools. The 
group moderation would first explore 
any general lack of matching between 
the overall teacher rating distribution 
and the overall distribution of resulu 
on the national tesu. The general aim 
would be to adjust the overall teacher 
raong results to match the ovenil re- 
sulu of the national tests; if the group 
were to have eltar and agreed reasons 
for not doing this, these should be 
reported. . . {andjdepartura could be 
approved if the group as a whole could 
be convinced that they were justified in 
particular cases.^ (Emphasu added) 



development that teachers need and de- 
sire. Both equity in testing and reform of 
schooling ultiinately depend on a more 
open and cooseosual process of establish- 
ing and upholding schodwide standards. 

A number of reasons are often cited 
for retaining "objective" tests (the design 
of which is usually quite "subjective^, 
among them: the unreliability of teacher- 
created tests and the subjectivity of hu- 
nun judgtneitt. However, reliability i'i 
only a problem when judges operate in 
private and without shared criteria. In 
fact, multiple judges, when properly 
trained to assess actual student perform- 
ance using agreed-upon criteria, display 
a high degree of inter-rater reliability. In 
the Connecticut foreign language test de- 
scribed above, on the thousands of stu- 
dent tests given, two judges using a four- 
point scoring system agreed on a sudent's 
score 85% of the time.x Criticisms of 
Advanced Placement exams that contain 
essay questions tisually focus on the cost 
of scoring, not on problems of inter-rater 
reliability. Inadequate testing technolo- 
gy is a red herring. The real problem 
standing in the way of developing more 
authentic assessment with collaborative 
standard-setting is the lack of will to in- 
vest the necessary time and money. 

True criterion-referenced tests and di- 
ploma requirements, though difficult to 
frame in performance standards, are es- 
sential for esublishing an effective and 
just education system. We must over- 
come the lazy habit of grading and scor- 
ing "on the curve" as a cheap way of set- 
ting and upholding standards. Such a 
practice is unrelated to any agreed-upon 



inteUecnial standards and can reveal only 
where students stand in relation to one 
another. It tells us nothing about where 
they ought to be. Moreover, snidents are 
left with only a letter or number - with 
nothing to learn ftom. 

Consider, too, that the bell-shaped 
curve is an intended result in designing 
a means of scoring a test, not some co- 
incidental statistical result of a mass test- 
ing. Norm-referenced tesu, be they lo- 
cally or nationally nomoed, operate un- 
der the assumption that teachers have no 
effect — or only a random effea - on 
students. 

There is nothing sacred about the 
Donnal curve. It is the distrihutioa most 
appropriate to chance and random ac- 
tivity. Education is a purposeful activi- 
ty, and we seek to have the studeou 
learn what we have to teach. . . . [W]e 
may even insist that our efforts are un- 
successful to the extent that the distri- 
bution of achievement appioximates 
the normal distribution.^' 

In addition, such scaring insures that, 
by design, at least half of the student 
population is always made to feel inept 
and discouraged about their work, while 
the other half often has a feeling of 
achievement that is iUusoty. 

Grading on a curve in the classroom 
is even less justifiable. There is no su- 
tistical validity to the practice, and it 
allows teachers to continually bypass the 
harder but more fruitfiil work of setting 
and teaching perfomunce criteria from 
which bener learning would follow. 

To let students show off what they 



At the school-site level in the U.S., we 
might consider the need for an oversight 
process akin to group moderation to in- 
sure that students are not subject to ec- 
centric testing and grading - a coomiit- 
tee on testing standards, for example. In 
short, what group tiMderation can pro- 

vide is the kind of on-going professional 2 2 
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know tnd are able to do is a very differ- 
ent business from the fatalism induced by 
counting etrors on contrived questions. 
Since standardized tests are designed to 
highlight di^erences, they often end up 
exaggeradng them (e.g., by throwing out 
pilot questions that eveiyone answers cor- 
rectly in order to gain a useful "spread" 
of scores).}* And since the tasks are de- 
signed around hidden and often arbitrary 
questions, we should not be surprised if 
tl)e test results end up too dependent on 
the native language ability or culniral 
background of the students, instead of on 
the fruit of their best effoiu. 

Tracking is the inevitable result of 
grading on a curve and thinking of stan- 
dards only in terms of drawing exag- 
gerated comparisons between students. 
Schools end up institutionalizing these 
differences, and, as the very word track 
implies, the standards for different tracks 
never converge. Smdents in the lower 
mcks are not uught and assessed in such 
a way that they become better enabled to 
close the gap between their current com- 
petence and ideal standards of perform- 
ance. » Tracking simply enables students 
in the lower tracks to get higher grades. 

In the performance areas, by contrast, 
high standards and the incentives for 
students are dear.^ Musicians and ath- 
letes have expen performeis constantly 
before them ftom which to learn. We set 
up different weight classes for wrestling 
competition, different rating classes for 
chess tournaments, and separate vzisivy 
and junior varsity athletic teams to nur- 
ture smdents' confidence as they slowly 
grow and develop their skills. We assume 
that progress toward higher levels is not 
only possible but is aided by such group- 
ings. 

The tangible sense of efficacy (aided 
by the desire to do well publicly and the 
power of positive peer pressure) that 
these extracurricular activities provide is 
a powerful incentive. Notice how often 
some smdents will uy to sneak back into 
school after cuning class to submit them- 
selves to the rigors of athledcs, debate, 
or band practice - even when they are 
not the surs or when their team has an 
abysmal record.'' 



CRITERIA OF AUTHENTICrrY 

Prom the arguments and examples 
above. k\ me move to a consideration of 
a set of criteria by which we might dis- 
tinguish authendc from inauthendc forms 
of testing." 



Structure and logistia. Authentic tests 
are more appropriately public, involving 
an actual audience, client, panel, and so 
on. The evaluation is typically based on 
judgment that involves multiple criteria 
(and sometimes multiple jtidges), and the 
judging is made reliable by agrmd-upon 
standards and prior training. 

Authendc tests do not rely on unrealis- 
tic and arbitrary time constraints, nor do 
they rely on secret questions or tasks. 
They tend to be like portfolios or a full 
season's schedule of games, and they em- 
phasize smdent progress toward mastery. 

Authentic testt require some collab- 
oradon with others. Most professional 
challenges faced by adults invdve the ca- 
pacity to balance individual and group 
achievement. Authentic tests recur, and 
they are worth practicing, rehearsing, 
and retaking. We become better educat- 
ed by taking the test over and over. Feed- 
back to students is central, and so audien- 
tic tests are more intimately connected 
with the aims, structures, schedules, and 
policies of schooling. 

InteUectual design features. Authendc 
tests are not needlessly intrusive, arbi- 
trary, or contrived merely for the sake 
of shaking out a single score or grade. 
Instead, they are "enabling" - construct- 
ed to point t^ ^ smdent toward more 
sophisticated and effective ways to use 
knowledge. The characteristics of com- 
petent performance by which we might 
son nonenabling from enabling tests 
might include: "The coherence of [the 
smdent's] knowledge, principled (as op- 
posed to merely algorithmic] problem 
solving, usable knowledge, attendon-free 
and e^icient performance, and seif-reg- 
ulatory skills."" 

Authendc tests are contextuaiized, com- 
plex intellectual challenges, not frag- 
mented and stauc bits or tasks. They cul- 
minate in the smdent's own research or 
produa, for which "content" is to be 
mastered as a means, not as an end. 
Authentic tests assess student habits and 
repenoires; they are not simply restrict- 
ed to recall and do not reflect lucky or 
unlucky one-shot responses. The port- 
folio is die appropriate model: the general 
task is to assess longimdinal control over 
the essendals.^ 

Authendc tests are representtdve chal- 
lenges within a given discipline. They are 
designed to emphasize realisnc (hut fair) 
complexity; diey stress depth more dian 
breadth. In doing so, they must neces- 
sarily involve somewhat ambiguous, ill- 
strucnired tasks or problems, and so they 



make student judgment central in posing, 
clarifying, and tackling problems. 

Standards of gnuUng and scoring. 
Authentic testt measure essenuals. not 
easily counted (but relauvely unimpor- 
tant) errors. Thus die criteria for scor- 
ing them must be equally complex, as in 
the cases of the primary-t rait s coring of 
essays or die scoring of ACHTL tests of 
foreign languages. Nor can audiendc tests 
be scored on a curve. They must be 
scored widi reference to audiendc stan- 
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dards of performance, which smdents 
must undersund to be inherent to suc- 
cessful performance. 

Authendc tests use muldfaceted scor- 
ing systems instead of a single aggregate 
gnde. The many variables of complex 
performance are disaggregated in judg- 
ing. Moreover, self-assessment becomes 
more central.^^ 

Authendc tests exist in harmony widi 
schoolwide aims; they embody standards 
to which everyone in the school can as- 
pire. This implies die need for school- 
wide policy-making bodies (other than 
academic departments) that cross discipli- 
nary boundaries and safeguard die essen- 
tial aims of the school. At Alvemo Col- 
lege in Milwaukee, all faculty members 
are bodi members of disciplinary depan- 
ments and of "competency groups" that 
span all departments. 

Fairness and equity. Rather dian rely 
on right/wrong answers, unfair "distrtc- 
tors," and odier stadsdcal ardfices to wid- 
en the spread of scores, audiendc tests 
ferret out and idendfy (perhaps hidden) 
strengths. The aim is to enable die stu- 
dents to show off what they can do. Au- 
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thendc tests strike a constantly eximined 
balance between honoring achievement, 
progress, native language skill, and pri- 
or fortunate training. In doing so, they 
can better reflect our intellectual values. 

Authentic tests minimize needless, un- 
fair, and demoralizing comparisons and 
do away with £ualistic thinking about re* 
sultt. lliey also allow q)propriate room 
to accommodate studenu' learning styltt, 
sptinides, and interests. There is room 
for the quiet "lechie" and the show-off 
prima donna in pUys; there is room for 
the slow, heavy lineman and for the 
small, fleet pass receiver in football. In 
professional work, too, there is room for 
choice and style in tasks, topics, and 
methodologies. Why must all students be 
tested in the same way and at the same 
time? Why should speed of recall be so 
well-rewaided and slow answering be so 
heavily penalized in conventional test- 
ing?J» 

Authentic tests can be - indeed, 
should be - attempted by all smdents, 
with the tests "scaffolded up," not 
'dumbed down" as necessary to compen- 
sate for poor skill, inexperience, or weak 
training. Those who use authentic tests 
should welcome student input and feed- 
back. The model here is the oral exam 
for graduate smdents, insuring that the 
student is given ample opportunity to :x- 
piain his or her work and respond to criti- 
cism as integral paiu of the assessment. 

In authentic testing, typical procedures 
of test design are reversed, and account- 
ability serves smdent learning. A model 
task is first specified. Then a fair and 
incentive-building plan for scoring is de- 
vised. Only then would reliability be con- 
sidered. (Far greater aittnuon is paid 



Only a humane 
and intellectually 
valid approach to 

evaluation can help 
us insure progress 
toward national 

intellectual fitness. 



throughout to the test's "face" and "eco- 
logical" validity.) 

As I said at the outset, we need a new 
philosophy of assessment in this country 
that never loses sight of the sudent. To 
build such an assessment, we need to re- 
turn to the roots of authentic assessment, 
the assessment of performance ofexeni' 
plary tasks. We might stan by adopting 
the manifesto in the intnxiucdon of the 
new national assessment report in Great 
Britain, a plan that places the interests of 
smdents and teachers first: 

Any system of assessment sboc!d 
satisfy goenl critem. For tbe purpose 
of national assessment we give priori- 
ty to the foUowing four cnteiit: 

• the assessment results should 
give direct infomsauon about pupils* 
achievement in relation to objectives: 
they should be critenon-referenced; 

• the results should provide t basis 
for decisions about pupils' further 
learning needs: they should be forma- 
tive; 

• the grades should be capable of 
comparison across classes and scbools 
... so the asseumenu should be 
calibrated or moderated; 

• the w«yi in which criteria are set 
up and used should relate to expected 
routes of eriiicanonal developmeot, giv- 
ing some continuity to a pupil's usess- 
ment at different ages: the assessmenu 
should relate to progression.*^ 

The task is to define reliable assess- 
ment in a different way, committing or 
reallocating the time and money needed 
to obtain more authentic and equiuble 
testt within schools. As the British pro- 
posals imply, the professionalization of 



teaching begins with the freedom and re- 
sponsibility to set and upheld clear, ap- 
propriate standards - a feat that is im- 
possible when tests are seen as onerous 
add-ons for "accounubility" and are de- 
signed externally (and in secret) or ad- 
ministered internally in the last few days 
of a semester or year. 

The redesign of testing is thus linked 
to the restrucmring of schools. The re- 
structuring must be built around intellec- 
nial standards, however, not just around 
issues involving govemanoe, as has too 
often been the case so fur. Authentic re- 
structuring depends on continuaUy ask- 
ing a series of questions: What new meth- 
ods, materials, and schedules are re- 
quired to test and teach habits of mind? 
What structures, incentives, and policies 
will insure that a school's standards will 
be known, reflected in teaching and test 
design, coherent schoolwide, and high 
enough but stiU reachable by most stu- 
dents? Who win monitor for teachers' 
failure to comply? And what response to 
such fulure is appropriate? How schools 
frame diploma requirementt, how the 
schedule supports a school's aims, how 
job descriptions are written, bow hiring 
is carried out, how syllabi and exams are 
designed, how the grading system rein- 
forces standards, and how teachers po- 
lice themselves are all inseparable {^m 
the reform of assessment. 

Authentic tests must come to be seen 
as so essential that they justify disrupt- 
ing the habitt and spending practices of 
conventional schoolkeeping. Otiierwise 
standards will simply be idealized, not 
made tangible. Nor is it "soft-beaited" to 
worry primarily about the interests of sni- 
denu and teachers: reform has little to do 
with pandering and everything to do with 
the requirements for effective learning 
and self-betterment. There are, of course, 
legitimate reasons for taking the intellec- 
nial pulse of students, schools, or school 
systems through standardized tests, par- 
ticularly when the resuiu are used as an 
"ancho^ for school-based assessment (as 
the British propose). But testing through 
matrix sampling and other less intrusive 
methods can and should be more often 
used. 

Only such a humane and intellectually 
valid approach to evalun'-'on can help us 
insure progress toward national inteUec- 
tual fitness. As long as we hold simplis- 
tic monitoring tests to be adequate models 
of and incentives for reaching our in- 
tellectual standards, student performance, 
teaching, and our thiiiking vxl discussion 
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As we abandon traditional views of education, 
the skills of thinicing and problem solving will 
replace discrete subject areas as the core of 
the curriculum and will lead to changes in 
instruction and assessment. 



Enterprising learning communities 
looking toward education in the 
next century know that new 
educational goals are imperative 
for our youth's survival, the 
continuance of our democratic institu- 
tions, and even for our planetary exis- 
tence. Learning how to learn 
throughout a lifetime; knowing how to 
behave when answers to complex 
problems are ambiguous, dichoto- 
mous, and paradoxical: and gener- 
ating, organizing, and applying an 
abundance of technological informa- 
tion are just a sample of the types of 
goals we need to establish. 

From such goals, we will establish 
world-class standards. But to do this, 
we must be prepared for a paradigm 
shift. We will have to replace some of 
our obsolete, traditional views of 
education with more modem, relevant, 
and consistent ones. We will let go of 
our obsession with content acquisition 
and knowledge retention as merely 
ends in themselves. We will dismiss 
uniformity and begin to value diver- 
sity. We will extinguish external eval- 
uation of students and teachers in 
favor of self-evaluation. We will 
replace extrinsic rewaids with learning 
activities that are intrinsically moti- 
vating. We will deflate competitive- 
ness to expand interdependence. We 
will redetlne smart to mean knowing 
how to draw fonh from a repenoire of 
strategies, knowledge, perceptions. 

Educational Le.adership 



and actions 
according to 
contextual 
demands. 

Since all 
aspects of a 
system are inter- 
locking, all parts 
must change in accordance with the 
new paradigm. No one part can 
operate efficiently unless the other 
parts of the system work harmoniously 
(Kuhn 1970). Imposing higher stan- 
dards, therefore, is not just "kid stuff." 
Higher standards must be set for all 
components of the educational enter- 
prise. We will need to evaluate the 
contribution of all parts of the system 
to higher standards in curriculum, 
instruction, and assessment 

Higher curriculum standards. Our 
obsession with the archaic compart- 
mentalization of the disciplines keeps 
school staffs separated. As intellectual 
development, thinking, problem 
solving, and cooperating become the 
core of the curriculum, traditional 
content and subject-matter boundaries 
will become increasingly obscure and 
selectively abandoned. Process will 
become the content of instruction. 

Our view of instruction will shift 
from learning of the content to 
learning yrom the content. History, 
physical sciences, mathematics, or the 
arts in the curriculum will no longer 
be ends in themselves; instead, 
problem-based instruction will serve 
as the vehicle for learners to integrate, 
reflect on. and transfer the unique 
knowledge, structure, and modes of 
inquiry of several disciplines. 
Teachers will select relevant, problem- 
centered, integrative themes because 
of their contributions to the thinking/ 
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learning process, and we will focus 
standards on applying concepts from a 
variety of fields to produce new knowl- 
edge, transfer strategies to novel situa- 
tions, and tackle complex problems. 

Higher instructional standards. 
Having worid-class standards will 
make us begin to see the profession of 
teaching as Intellecnially complex, 
collaborative, and reflective. Teachers 
who have achieved the highest stages 
of intellectual functioning are more 
commined to and empathk with indi- 
vidual students, and they provide 
greater sdmulation for students to 
function at higher levels of cognitive 
complexity as well (Glickman 1985). 

In traditional school settings, time 
limits, isolation, and minimal peer 
interaction prevent teachers of 
different departments, grade levels, 
and disciplines from meeting together. 
Thus, teachers' intellectual growth is 
diiTunished. Teaching toward worid- 
class standards will require mutual 
interaction from teachers who work 
with students from various age levels 
and possess rich funds of knowledge 
in various fields. New roles for 
members of instructional teams will be: 

■ knowledge managers who judi- 
ciously select topics and problems for 
their contribution to achieving these 
standards: 

■ team planners who continually 
clarify the school community's desired 
outcomes and create instructional 
strategies to achieve their goals: 

■ collegial coaches who teach, 
observe, and give nonjudgmental feed- 
back to one another 

■ collaborative researchers who 
experiment with and evaluate 
curriculum and instructional effective- 
ness and modify them accordingly. 




Higher assessment standards. We 
cannot employ traditional product- 
oriented assessment techniques to 
evaluate the achievement of these new, 
process-oriented standards. Skillful 
teaching teams and students them- 
selves (who are the best collectors of 
assessment data) will, over time, 
observe, record, interpret, and report 
evidence of growth toward and 
achievement of these standards. Such 
assessment strategies will include: 

■ directly observing performance 
in collaborative problem-solving 
situations: 

■ collecting logs, journals, and port- 



folios of selected artifacts of learning 
excellence; 

■ observing performances while 
conducting extended cooperative 
projects; 

■ conducting interviews to discover 
students' self-perceptions as problem 
solvers; 

■ maintaining checklists recording 
indicators of growth toward desirable 
habits of mind; 

■ assessing displays, exhibitions, 
and performances according to both 
internal and external criteria; and 

■ employing media and advanced 
technology to assist in collecting and 



of society, will achieve higher, world- 
class standards only when American 
society imposes higher, world-class 
standards on itself. Achieving higher 
standards requires the devotion of the 
greatest share of our resources to the 
development of each person's fullest 
potential. ■ 
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INNOVATIONS 



Smart Tests 



A new approach to assessment helps teachers 
understand the ways young children think and grow. 



Whtn Mary Ann 
Cockman mIu htr 
kindergarten pu- 
pils to draw talf- 
portrait* at tha 
surt of tha school 
year, they often 
sketch e lerge 
head, two eyes, and a stick for a 
body. When she asks them to do 
the same thing later in the year, 
their renderings become more de- 
acquiring ears, mouth, 
3stick, eyebrows, eye- 
L. fingers, shoes, recten- 
gular legs — even designs on their 
clothing. 

Cockman, who teaches at 
Thomson Elementary School in 
Davidson, Mich., says the student 
portfolios she now collects to doc- 



ument such progrtsa— aa well aa 
tha detailed checklist* and writ- 
tan evaluations she prtparasaev- 
eral time* a year on ceeh child- 
have helped her become more 
attuned to the ways young chil- 
dren think and grow. *You learn 
to look at children difTenntly,* 
*h**ayt. 

Thomson Elementary is one of 
37 schools nationwide piloting a 
system of sssessment designed to 
provide en eltamative to stan- 
dardized test* and traditional re- 
port card* in the early gredes. 
The approach, known as the 
*Work'SampIing System,* wu de- 
veloped by Samuel Meisele,apro- 
feasor of education end an associ- 
ate deen for reseerch at the 
University ofMichigen. 



Meisels is one among many ex- 
pert* who argua that traditional 
*tandardix*d taata ar* uaraliabl* 
for young children and hava beea - 
used inappropriately to delar 
achool entry, retain large num- 
bers of students, and justify rote- 
skill instruction. Meisels' goal is 
to replace group-administered 
echievementtesting with teacher- 
focused performance assess- 
ment. Such e change, he says, 
would improve both teeching 
practices and children'* experi- 
ences in the eer'./ grades. ' 

Tesehers using the Work Sam- 
pling System complete a detailed 
eheckliat three time* a year track- 
ing children'* perfonnence in 
seven domains: personal and so- 
cial development, language and 



23 

BEST COPY AVAILABLE 



literacy, methemetieal thinking, 
scientirw thinking,aocial and cul- 
tural awareness, art end music, 
end physiea) developmant They 
, alsaeellectsampleaorehildrtn'e 
work in eech demein, including 
taaks asked of ell childnn end 
Individual itame thet very by 
child. 

Then, using the checkliets, port- 
foliea, e nd other observation*, the 
tmcheis templet* a "summery re- 
port," or student prenie, evel- 
ueting e child'e ptrformanct, 
eirengths, and didiculties in eech 
domain. The** report*, which are 
ela* prepared three time* a yeer, 
er* deaigned to Mrv* aa a beai* 
for communicating with other 
teachers, perents, and adminis- 
trators and for planning student 
instruction. 

Initial studies to detarmine 
whether outside experts would 
rate children the same way their 
claseroom teachers do based on 
the portfolio* and checkliats have 
shown 'extremely high" reliabil- 
ity, according to Meisels. And a 
etudy designed to gauge the validr 
ity of the a**e«*ment compared 
with an individually admini- 
*t*f*d nonn-r*f*r*nc*d *tand*rd- 
Ixed test, he *ay*, eleo produced 
promising reaults. 

Still, Meisels cautions thot the 
Work Sempling Syetem ehould 
not be used as a "gatekeeping 
device" for Khool entry or promo- 
tion. And while data derived from 
tha system could be used to com- 
pare the perfomenee of cless- 
rooms or large group*, h* *ay*, 
ita chief purpoee i* to aa**** indi- 
vidual children'e progrca* over 
time, gauge eehiavement egainat 
curriculum goal*, .and tailor in- 
*truction to children'* need*. 

Lorri* Sheperd, * profe**or of 
education at tiw University of Col- 
orado and author of etudiee on 
the adverse elTects of retention in 
the esriy grades, ssy* the biggest 
danger in introducing altcmati va 
early childhood es****menta oc- 
curs "when people take a meea- 
ure dceigned for one purpoae and 
retrofit it for another." Much 
mora experimentation and 
fieldwork in the higher grades 
ere needed, ehe maintaine, to 
reliebly use such data to rate the 
quelity of teachers and schools. 
The kinde of preliminary data 
we've seen are sulTieient to jus- 
tify classroom uses of these on- 
sessmenU," shs ssya, "but would 
not be sufTicient for external ac- 
countnbility purposes." 

Such fl**e**ments srs pcrheps 
most useful in helping teachers 
leem more about child develop- 
ment JoAnne Lowe, e kindergar- 
ten teacher at Copelend Ele- 
mentary, a pilot Khool in Dexter, 
Mich., ssys the Work Sampling 
System hes hstped her become a 
better obeerver of the whole 
gamut of children'e needs — in- 
cluding thsirphysical growth and 
personal and social develop- 
ment — and has "made me aware 
oferees I waan't strong at." 

"One of the best things about it 
ie that children do not full," any* 
Janice Bmwn, prindpnl oTKrUar- 



ing Elementary in Willow Rt 
Mich., another perticipeti 
school. "Whst you ar* measur 
a child ageinat ie him- or htm 
not e felse presumption like 
grade or tasL If no growth i* e 
dent, then ws can leek at it a 
look at how we are going 
diange our system." 

One drawback, teacher* aay 
that tha system is very 'in 
eenauming. But, they add. p 
ents liks ths detailed feedbr 
this kind of assessmsnt ci« 
thsm,and ttbuoyschildr*n'ss< 
ssteem. "Kids, by the end of t 
year, ere able to look at th 
portfolio and eee their o< 
growth," says Penny Butler, a k 
dergarten teacher at Gariield E 
mentaiy School in Flint, Mich. 

While they appteud the level 
training and technical assistar 



%eoftlie best 
tilings about it is 
that cliildren do 
not fail,' says om 
piincipal. 'Wliat 
you are measui'iu 
a cliild against is 
liiin- or herself, 
not a false 
presumption hke 
a grade or test.' 



provided by Meisels and his cc 
Isaguee, soma educators still pr 
diet It will Uka a few years fi 
them to become comfortab' 
uaing the system. And Harvai 
University psychologist Hewsi 
Gardnsr notes thst it also wi 
take time to "develop canon* < 
looking *nd li*tening"— Ilka the; 
eatabliahed to rate performem 
In other field*— that pol 
cymakets would conaidcr acctp 
able for accountability purpoee* 
S*y* Principnl Janice Brawi 
"One of the herd question* cduci 
tor* have to anawer is: Ar* w 
interested in knowing how chi 
dren compere with on* nnolhe 
or ere we intereated in knewin 
how much children ar* leamin( 
If we heva decided wa ar* mor 
interested In knowing hew muc 
children are leeming, wemuetd 
aemathing Ilka tills." ■ 

— Dabornli !«. Cohai 
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Laser Disk Portfolios: 
Total Child Assessment 



Multimedia technology has transformed 

the assessment process for students and teachers 

at a rural elementary school in Wyoming. 



Jo Campbell 
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As this story begins, a 4th grade 
student is sitting at a multi* 
media system consisting of a 
computer, CD-ROM drive, optical 
drive, scanner, and laser printer. The 
student has requested that a story he 
wrote become part of his permanent 
assessment record. He is scanning his 
work onto his laser disk. As he works, 
another student asks to take the video 
camera to the art room. She wants to 
record a video of herself throwing a 
pot on the potter's wheel as one of her 
assessments for the year. When she 
finishes, she returns to the multimedia 
system to transfer the video to her 
laser disk. 

Earlier today, a student has used the 
video camera to record the play in 
which he had the leading role. 
Another student has documented on 
disk that she has the ability to climb 
the rope in physical education class. 

All of this is happening at 
Conestoga Elementary School in 
rural Wyoming, where the idea of 
creating portfolios of students' work 
as a means of assessment has been 
combined with laser disk technology. 
The laser disk portfolio assessment 
system is based on the work of IBM 
consultants and researchers from 
Project Zero at Harvard. A grant from 
the Wyoming State Department of 
Education Super School Program 
helped our school purchase the neces- 
sary IBM hardware. While the tech- 
nical implementation has had its 
frustrating moments, the IBM 
consultants and school staff have 
persevered. 
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Using Laser Disk Assessments 
Students and teachers at Conestoga 
have begun to use the laser disk port- 
folio assessment system this year. 
Large amounts of infoimation can be 
added to or retrieved from the system 
as many times as necessary. Yet the 
laser disks are so small they can slide 
into any student's permanent file, 
eliminating the need to find additional 
filing space. 

Teachers can use the system to 
research their classes before the first 
day of school. The system helps them 
do something as simple as putting 
students' names and faces together, or 
something as complicated as deciding 
what teaching and learning activities 
would best increase outcomes for the 
students. 

To begin planning assessment, 
grade-level teams determine criteria to 
evaluate growth for each child. 
Teachers do a preassessment the first 
month of school by videotaping their 
students. These preassessment videos 
might record a student playing a game 
with another student, reading a story 
of the student's choice, or answering a 
few questions as the video camera 
records the student's ability to talk 
with an adult. 

Teachers plan to use the system for 
formal assessments at least twice a 
year with each student. Has the 
student gained reading fluency? Has 
handwriting improved? How else has 
the child grown? Questions such as 
these can be answered by reviewing 
the disk. 

During the school year, teachers can 
review the disks of students who are 
sUTJggling with schoolwqrk tadeter- 
mine" what the students' interests are 
and in what areas they have excelled 
previously. The disks can be used to 
motivate the child who says, "1 can't 
do it," by showing how much he or 
she has done over the years. Self- 



esteem builds as the accomplishments 
of each child are recorded and as 
growth is measured against individual 
standards, rather than group standards. 

At the end of the school year, an 
annual ritual of passage will take place 
as each 6th grader and his or her 
parents, along with the teacher and 
principal, review the student's disk 
from kindergarten to 6th grade. We 
are expecting tears and laughter and 
possibly embarrassment from students 
viewing their antics of years gone by, 
but we are also predicting that every 
family will want a copy of the disk for 
a personal history. In any event, the 
school will keep a disk for its records. 

Implementing the System 

A school planning team handled the 
implementation of Conestoga's laser 




disk portfolio assessment system. The 
counselor made home visits to talk to 
parents about the system and ascertain 
any concerns or ideas about assessing 
in this manner. Implementing the 
complete system with just one class 
the first year allowed staff participants 
to acquaint themselves with the 
system and become comfortable with 
the equipment before launching into 
assessments of the total school 
population. 

The kindergarten class will be the 
first class to be followed through the 
grades. This year we videotaped the 
kindergarmers answering predeter- 
mined questions and transfentd that 
image to their laser disks. Teachers 
also scanned the children's drawings 
onto the disks. j 

Getting a Complete Picture ^ 

Portfolio assessment is not a new 
concept in education, but a system 
that allows permanent storage of 
optical data, written and drawn 
images, and verbal ability is new. The 
staff and parents at Conestoga are 
excited about this new assessment 
system, especially as it relates to 
increasing student self-esteem. 
Recognizing that child development 
involves more than cognitive growth 
is the underiying belief behind the 
system. The students, parents, and 
staff will be looking at the laser disk 
portfolio assessments to determine 
growth in verbal ability, physical 
accomplishment, artistic achieve- 
ment, and self-assurance. □ 

Copyright © 1992 by Jo Campbell. 



Jo Campbell is Principai of Conestoga 
Elementary School and a doctoral candi- 
date at Teachers College, Columbia 
University, New York City. She can be 
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4901 Sleepy Hollow Blvd., Gillette, WY 
82716. 
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Portfolios Invite Reflection 
from Students and Staff 



Learning Experience Forms and Portfolio Evenings 
are just two of the ways Crow Island School found 
to make the assessment process more meaningful. 

Elizabeth a. Hebert 



Four years ago. Crow Island 
Elementary School began a 
project that has reaped benefits 
far beyond what any of us could have 
imagined. The focus of the project was 
assessment of children's learning, and 
the tangible product is a new reporting 
form augmented by student portfolios. 

More important, however, has been 
tlie process of developing our thinking 
and teaching around new ways of 
looking at children's learning. In fact, 
this process became more valuable to 
us as a faculty than the assessment 
product, helpful as it has been. 

Our Commitment to 
Alternative Assessment 

The project grew out of our dissatis- 
faction and frustration with mandated 
standardized modes of assessment. 
Standardized tests do not reflect how 
we teach, the ef fects of our teaching 
on children, or how we adapt instruc- 
tion to individual learners. Wolf and 
colleagues write that "the design and 
implemenialion of alternative modes 
of assessment will entail nothing less 
than a wholesale transition from what 
we call a te.uins culture to an assess- 
ment culture." They continue: 

The observable differences in the 
form, tiie data, and the conduct of 
standardized testing and its alterna- 
tives are in no way superficial 



matters or mere surface features. 
They derive from radical differ- 
ences; in underlying conceptions of 
mind and of the evaluation process 
itself. Until we understand these 
differences and their network of. 
consequences, we cannot develop 
new tools that will allow us to 
ensure that a wide range of students 
use their minds well (1991, p. 33). 

Obviously, we had our work cut out 
for us. What did we do to reatTirm our 
commitment to a concept of learning 
incompatible with .standardized 
testing? First, we did a gixxJ deal of 
reading; engaged in lengthy discus- 
sions about values, community 
building, and conferencing; and 
consulted with experts. We also 
became more deliberate about making 
time to visit one another's classrooms 
and to share and refine our observa- 
tions uf children. Next we began 
defming the questions to which we 
were seeking answers. Our first ques- 
tions were global: 

• How do we define learning? 

• Where does learning take place? 

• How do we recognize learning? 

• How do we report instances of 
learning? 

As we answf red these larger ques- 
tions, our concerns became more 
specific. How can we communicate 
about children's learning experiences 
with parents in ways that: 



• authentically describe the child, 

• speak to issues of accountability 
and maintain the integrity of our 
beliefs about children and how they 
learn, 

• reflect the different ways that 
teachers organize instruction, 

• provide concrete information 
compatible with parents' expectations? 

A Compatible Theory 

Some background information about 
our school provides a context for our 
project and how we went about 
answering these questions. Crow 
island is a public JK-S school in 
Winnetka, Illinois, an affluent suburb 
of Chicago's north shore. The 
Winnetka Public Schools include «hrce 
elementary schools and one middle 
school for grades 6-8. Our lower 
schools have enrollments of 360-390. 
Although a public school system, we 
have a strong tradition in the progres- 
sive philosophy of education that is 
distinguished by: 

• a commitment to a developmental 
orientation to instruction, 

• the priority placed on considera- 
tion of the "whole child" and his or 
her individual mode of leaniiiig, 

• the absence of letter grades until 
7th grade, 

• high regard for teachers as profes- 
sionals. 

In acknowledging the uniqueness of 
a child's mode of learning, the district 
has placed a high priority on confer- 
encing with parents. For many years, 
pupil progress has been reported to 
parents in a conference format three 
times per year. Teachers had prepare;' 
narrative descriptions of children 
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using the following organizers: 
language arts, math, social studies, 
science, growth of the child as a 
leantcr, and growth of the child as a 
group menttx;r. 

One expert who influenced our 
thinking about alternative assessment 
was Howard Gardner, whose "Tlieory 
of Multiple Intelligences" (musical, 
linguistic, logical-mathematical, 
spatial, bodily-kinesthetic, interper- 
sonal, and intrapersonal) challenges 
the more traditional concepts of intel- 
ligence. The main thrust of Gardner's 
theory as applied to schools is that 
children may demonstrate the different 
kinds of intelligences in ways not 
necessarily associated with traditional 
school subjects and certainly not asso- 
ciated with traditional modes of 
assessment. Gardner's theory 
resonated with the themes of progres- 
sive education to which we at Crow 
Island are devoted. 

A Visual Format 

Gardner's theory provided a good 
scaffold for our thinking. The next 
step was to put our thoughts into a 
visual format. Our first rough attempt 
began to capture the idea of multiple 
dimensions of a child's learning. This 
primitive model consisted of a stick 
figure surrounded by floating boxes. 
As you may expect, there was much 
discussion about the number, size, and 
positioning of 'he boxes, but we 
finally agreed on a format. We call it 
our Learning Experiences Form (see 
fig. I for a composite example of the 
form, shortened for space). 

Our next concern was to identify 
our organizers on the Learning Experi- 
ences Form. Being committed to the 
multiple intelligencer* perspective, we 
readily included music, art, and phys- 
ical education. We wanted to recog- 
nize these teachers' long-term rela- 
V. )nships with students, the value of 



their programs, and their insights 
about children's learning. But what 
about the other Learning Experience 
organizers? The dialogue went some- 
thing like this: 

Q: How should I specify my 
organizers? 

A: That depends on how you 
organize instruction. 

Q: But what if mine are different 
from someone else's? 

A: That's OK. You organize 
instruction differently. We already 
know that about one another. Now 
we're just writing about it. 

Q: But we organize instruction 
dilTercnily lor different .students, 

A. Your Learning Experiences 
Form will then reflect the flexibility 
of your teaching. 

This was a cnicial stage in our 
thinking because discussing the form 
brought to the surface what I term the 
"bilingualism" of teachers. Inside 
langtui({t' — what we do in our class- 



rooms — reflects our beliefs and 
values, years of teaching experience, 
observations of children and of other 
good teachers, and confidence in 
knowing what we know. Outside 
langtuige — what we say we do in our 
classrooms — is influenced by 
community values, comfort level 
within the school environment, polit- 
ical pressures, district and administra- 
tive policies, test scores, and 
curriculum. 

The nature of our project necessi- 
tated our speaking "inside language," 
a more difficult discourse because it 
requires feelings of safety and, secu- 
rity. Gradually, though, we were able 
to experience the sharing of values 
that leads to the creation of a secure, 
thoughtful environment forchildren,| 
teachers, and parents. 

A Close Look at Ourselves 

In order to change how we evaluated 
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children's learning, we real- 
ized we needed to lake a 
close look at ourselves. We 
soon found ourselves 
undergoing an intensive 
assessment of our teaching, 
our beliefs about children, 
and our views of the school 
and its relationship to our 
community. 

At this point, the project 
quite naturally proceeded 
from an emphasis on 
student assessment to a 
more powerful staff devel- 
opment focus. In order for 
this to occur in any school, 
administrators must commit 
to providing tJie kind of 
school environment where 
^uch a climate can flourish. 
Administrators also have to 
acknowledge that ail 
teachers do not arrive at the 
same point in their growth 
together. As we emphasize 
with the children, teachers 
must construct their own 
knowledge of children, how they learn, 
and how to evaluate that learning. We 
have to be patient and sufi'icientiy open 
to allow for different stages of under- 
standing, yet focused enough to 
provide clarity and vision to the effort. 

Improvements to the Process 

We began using the Learning Experi- 
ences Form in a variety of ways. Some 
teachers were more conservadve, 
using traditional school subject labels 
on their forms. Others coined new 
organizers that reflected their teaching 
styles. As they struggled with the new 
'"ormat, teachers became more 
.liouglurul; and parents, sensing liie 
positive energy and concern of 
teachers, responded enthusiastically. 
After the first conference using the 
^j, new form, Ihe response from both 
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parents and teachers was overwhelm- 
ingly positive. 

Over four years, we've refined the 
form to meet the suggestions of 
teachers at kindergarten, primary, and 
intermediate levels. In response to our 
concern about how to separate out 
curriculum .specifics and descriptions of 
a child's learning, one of our teachers 
designed a Curriculum Overview, to be 
printed on the back of the form, that 
consi.sts of miiii-.stuteinents of 
curriculum objectives for that portion 
of the year. Tlus addition freed up tlie 
front of the form for more focused 
descriptions of children's learning. 

Noting the absence of the child's 
input to the form, we designated a 
space for a "child's reflection" about 
his or her learning. Tlic older .students 
write their own thoughts; teachers take 

33 



dictation for the 1st graders. We've 
also begun to include parents' 
thoughts about their child's learning 
experience in our assessment form. 

Students Tell Their Stories 

The next step was to have our .students 
create portfolios. Portfolios are 
compatible with Crow Island's agenda 
for effective teaching, authentic 
a.sse.ssment, and faculty growth. One 
of the bc.st definitions in the current 
literature comes from Paui.son and 
Paulson (1991): "Portfolios tell a 
.story ... put in anything that helps tell 
the .story." Willi these authors, we al.so 
agree about the importance of the 
child's participation in selecting the 
contents of the portfolio and with a 
f(x:us more on process than on content 
(1991, p. I). 



At present, each of our students has a 
portfolio that represents work across all 
domains. Students maintain their port- 
folios all year and frequently have 
conferences with the teacher about 
works in progress, additions, and dele- 
tions. At the end of the year, their port- 
folios are combined with past years' 
work and stored in our Student 
Archives. The archives are alphabeti- 
cally arranged in open shelving in our 
Resource Center along with historical 
documents, publications, and 
photographs of our school and students. 

Portfolio Evenings 

Three years ago we added a new 
element to our assessment project. 
Encouraged by the kinds of thinking 
that children had expressed in their 
Student Reflections, we realized that 
they were capable of much more. 
Getting them more involved in the 
process of assessment seemed to make 
good sense: 

In preparation for "Portfolio 
Evenings," children review their port- 
folio/archive as teachers guide them 
with questions like: 

• How has your writing changed 
since last year (or since September)? 

• What do you know about numbers 
now that you didn't know in 
September? 

• Let's compare a page from a book 
you were reading last year and a book 
you are reading now and include 
copies of each in your portfolio (an 
idea from Denise Levine. Fordham 
University, New York). 

• What is unique about your port- 
folio? 

• What would you like Mom and 
Dud to understand about your port- 
folio? Can you organize it so it will 
show that? 

The idea is to ask guiding questions 
that help children reflect on their 
r»V/^ inking- Students are encouraged to 



write about their learning and to 
include these thoughts as part of their 
portfolios. Developing the metacogni- 
tive process in students, even at a 
young age, heightens their awareness 
and commitment to a critical assess- 
ment of their learning. 

In preparation for Portfolio 
Evenings, the teacher divides the class 
into small groups of six or seven at the 
primary level (and larger groups at 
grades 4 and 5) and assigns a night for 
each group of students and their 
parents. Primary-level Portfolio 
Evenings are held in February. We 
hold intermediate-level Portfolio 
Evenings in May, because older 
students prepare more extensive 
projects. 

On Portfolio Evenings, which last 
for about an hour and a half, the chil- 
dren sit with their parents and present 
their portfolios. The teacher and I 
circulate, visiting each student and 
highlighting particular milestones each 
youngster may have attained. We are 
available for questions but try not to 
intrude, because this is really the chil- 
dren's evening, and they need to "run 
the show" as much as possible. Parents 
and teachers have been impressed with 
the leadership and independence that 
even our youngest students have 
demonstrated in this setting. 

A Powerful Learning Experience 

We are continuing to refme our assess- 
ment project. Some issues we're 
addressing are practical in nature, for 
example, storage containers for the 
portfolios/archives. Others are more 
fundamental, like how to use portfo- 
lios to link children's early, strong 
expressions of interest in a particular 
topic to more sophisticated elabora- 
tions later in their school careers. We 
are also contemplating how to gain the 
community's support for these alterna- 
tive modes of assessment as part of a 
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viable system of accountability. And, 
finally, as a faculty we are trying to 
preserve the cohesive and bold spirit 
that nurtured this project along its way. 

The entire process has been a 
powerful learning experience for our 
faculty as well as for the children and 
their parents. It has expressed the 
fundamental values of our school 
district and represents our joint explo- 
ration of the complex issues of chil- 
dren and their teaming. We are 
encouraged to go forward by the posi- 
tive effects this project has had on the 
self-esteem and professionalism of the 
individual teachers and the inevitable 
strengthening of the professional 
atmosphere of the entire school. We 
have improved our ability to assess 
student learning. Equally important, 
we have become, together, a more 
empowered, effective faculty. □ 
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ORTFOLIO ASSESSMENT 

IN THE HANDS OF TEACHERS 

Skills and Training Needed To Meet a 
High-Stakes Challenge 



BY CLARE FORSETH 
Teacher. Marian Cross Public School, Norwich. Vermont 





TTu definition of 'assessment ' is broader than tests. In time, educators may realize that 'assessment " includes every activity that 
bToa [Kw mformatton about students. 




hose who shape educational 
policv in Vermont are engaged 
in a formidable task: a statewide 
innovative assessment of mathematics 
and writing ability of fourth and 
eighth graders. 

."^fter Vermont's legislature funded 
a program in 1990 that allowed the 
state's Department of Education to in- 



vestigate various assessment systems, a 
committee of teachers and consul- 
tants looked at what the current 
methods of assessment were measur- 
ing. The committee decided the long- 
used standardized test was not assess- 
ing many of the educational out- 
comes Vermonters value most. 
These policymakers then mrned to 
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"To understand... 
teachers heed tolD( 
probliii&lveii 




process 
approach to 
writing instruc- 
tion, which has 
been in place for more 
than a decade. In this ap- 
proach, student writers de- 
velop their ideas through many 
drafts, Mfith less emphasis on the 
end product than on the learning and 
thinking process that creates an in- 
creasingly precise refinement of ideas. 




Vermont 's commitment to po/rtfolio assessment requires teachers to 
become more than familiar with a new assessment tooL It also 
demands new instructional techniques. 



a portfolio assessment system that 
looks at student work and assesses it 
against established criteria. Unlike 
standardized test results, which reveal 
what a student does and does not 
know on one specific occasion, the 
portfolio showcases the student's best 
work over time. 

This alternative means of assess- 
ment represents a high-stakes chal- 
lenge for the state's teachers, asking 
much more of them than mere famil- 
iarit\- with a new assessment tool. Suc- 
cessful implementation of this form 
of assessment calls on teachers to 
make significant changes in their in- 
structional techniques. 



Sixth-grade 
students from 
Marion Cross 
Public School. 
.\'orunch. Vt., 
are solving a 
mathematical 
riddle in which 
they try to find a 
number that 
could be made 
into a souare 
and a cube. 



Process Approach 
To effect such changes, teachers need 
a \-anecy of support systems to develop 
their understanding of portfolio as- 
sessment and the new skills that sup- 
port it. Vermont has established sup- 
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port svstems to develop portfolio as- 
sessment skills in mathematics and 
writing. The change from other 
assessment methods is most clearly 
demarcated in the mathematics port- 
folio assessment. 
.\Ianv teachers are familiar with the 
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During much of this same period, 
though, mathematics instruction con- 
tinued to be driven by standardized 
tests that emphasized compuuuo 
and one-step word problems. Th' 
most widely used textbooks also re- 
flected these emphases. Unfortu- 
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natcly, these systems continued the 
stress on successful computation and 
correct answers that has long been the 
focus of the mathematics classroom, 
and this focus retarded the develop- 
ment of processoriented approaches 
to teaching mathematics. 

Vermont's investigation of various 
assessment systems convinced plan- 
ners of the importance of a more 
thoughtful analysis of which mathe- 
matical skills and abilities are truly 
valuable and lasting, and how these 
abilities might best be fostered 
through instruction and assessment. 
They recognized the need to begin to 
focus on problem solving and on the 
students' ability to communicate their 
reasoning. 

So significant a change in approach 
to assessment, combined with neces- 
sary changes in instructional ap- 
proach, could not be implemented 
overnight, even in a small state like 
Vermont. Teachers first had to learn 
just what was to be evaluated in port- 
folio assessment, and how. 

Evaluation Criteria 
In the portfolio assessment of mathe- 
matics, student work is used to assess 
problem solving and communication 
skills using these seven criteria: 

• understanding the task, 

• applying the jtrategy, 

• making decisions, 

• vending the solution, 

• making connections. 

• using rich mathematics lan- 
guage, and 

• using effective mathematical rep- 
resentation. 

The first task. then, was to familiar- 
ize teachers with these criteria. 

To this end. a committee of teach- 
ers developed institutes to introduce 
participants to the problem solving 
and communication criteria used to 
assess student work. Teachers re- 
sponded eagerlv to the chance to 
make meaningful changes in their ap- 
proach to teaching mathematics. 

One participant in a portfolio insti- 
tute said: "I feel both excited and chal- 
lenged by the mathematics portfolio 
project we are undertaking as a state. 
It is exciting to examine alternative as- 
sessment measures and challenging to 
confront the reality that this assess- 
ment project is driving a change in 
our way of providing mathematics in- 



struction to students. The mathemat- 
ics portfolio requires some dramatic 
shifts in our teaching. Teachers who 
have traditionally used drill activities 
and worksheets will need to rethink 
their instruction in order to ensure 
that students become effective prob- 
lem solvers, hence meeting the crite- 
ria set by the portfolio project" 

To understand the process their stu- 
dents go through as tiiey solve a prob- 
lem, teachers need to become prob- 
lem solvers themselves. During the 
institutes, teachers not only learn to 
assess student work, they abo are ac- 
tively engaged in problem solving. 
Sample problem-solving situations are 
presented to the teachers. One day 
teachers use tangram pieces to investi- 
gate fractions. Another day they use 
Cuisenaire rods to introduce the con- 
cept of ratio and proportion. In each 



portfolio institute come to under- 
stand problem solving involves more 
than simply getting the answer. It also 
encompasses communicating through 
mathematical language and a repre- 
sentation of how the student soWed the 
problem, why certain decisions were 
made along the way, and what connec- 
tions the student was able to make dur- 
ing the problem-solving process. 

An important goal of the institutes 
is for teachers to convey this aware- 
ness of the problem-solving process to 
students in their mathematics classes. 
This means involving students in 
many problem-solving situations and 
assessing the work using problem- 
solving and communication criteria. 

Through the assessment, students 
begin to vmderstand how to improve 
their solutions or presentations. Stu- 
dents who take ownership of the crite- 




Vermont trains tls teachers to serve as assessors. Each year they will evaluate a sample of portfolios 
from around the state against seven criteria to prepare a detailed analysis of student progress. 
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case the teachers work in cooperative 
groups and write up and share their 
strategy and reasoning with other par- 
ticipants. This type of model lesson al- 
lows the teacher to see exemplary in- 
structional techniques as they ex- 
perience being problem solvers. 

Student Ownership 

Teachers who have participated in a 
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ria begin to be able to assess their own 
work. They begin to focus on aspects 
of their presentation that are weak and 
strive to improve upon them. 

To provide students with rich prob- 
lem-solving tasks, teachers are finding 
they need a variety of resource mate- 
rials to supplement their textbook 
presentation of content areas. Teach- 
ers realize they do not have to rely ex- 




Studtntsin 
Ji/Agrade 
art calculat- 
ing jracxions 
using 
taHgtun 
pitces. 

Students are 
asktdto 
dtfendtheir 
dtdsiom. 



''Administratoi^^;^ 
: always rjeed toll 
be involved with'l 
:-theirteache^1 
and sensitive to/ 
their needsi;^^^' 



clusively on a textbook but can use 
the text as one of many resources. 

Problem solving cannot be an add- 
on to the curriculum. It needs to be 
an integral part of instrucuon. Teach- 
ers need to change the emphasis of 
instrucuon from computation and 
one-step word problems to complex 
mathematical tasks that engage stu- 
dents, tasks for which the student may 
not have a readily available strategy. 
Teachers are finding that introducing 
concepts through manipulatives and 
group work allows students to investi- 
gate mathematical ideas more thor- 
oughly than memorizing rules given 
to them by the teacher. 

Powerful Tool 

Teachers quickly realize just how pow- 
erful a tool the student's portfolio is. 
It presents a wonderful profile of the 
student's evolution as a mathemati- 
cian. It also provides an evaluation of 
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various aspects of the teacher's pro- 
gram, including instrucdonal oppor- 
tunities, content areas, and student 
empowerment. 

Rrst, the portfolio of student work 
should show that the student was given 
a variety of instructional opportuni- 
des. These opportunides should in- 
clude integrating problem solving and 
technology, interdisciplinary work, 
group work, use of manipuladves, and 
real-world applicadons. Teachers must 
acquire skills needed to present these 
opportunides in their classrooms. 

A second aspect the portfolio will 
reveal is that problem solving was not 
merely an add-on to the curriailiun, a 
twoweek lesson at some point during 
the year. Problem solving and investi- 
gations need to be integrated into all 
the content areas, and the student's 
portfolio should demonstrate this 
integration. 

Third, the development of mathe- 
matical power will require teachen to 
provide a classroom environment that 
encourages problem solving and 
models the curiosity, flexibility, and 
reflection we must instill in students. 
As students confront problems they 
have no ready means to solve, they 
begin to exhibit their power as math- 
ematicians. They become risk-takers 
who persevere through challenging 
and engaging tasks. They prove they 
are curious about mathematics, flexi- 
ble in approaching problems, and 
even able to reflect about their 
growth as mathematicians. 
Having one's program evaluated 
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through portfolio assessment — that is, 
in terms of the progress made by one s 
students on complex tasks — can se^j^ 
threatening. As teachers grasp Jf/jf 
ramifications of this new assessment 
program, they realize administrators 
have a crudal role to play in providing 
strong and positive support. 

Crucial Support 

Administrators always need to be in- 
volved with their teachers and sensitive 
to their needs, but this supportive en- 
gagement is especially important when 
major instructional changes are being 
implemented. The admmistrator will 
be asked to support the teacher in in- 
forming school board members, par- 
ents, and the community about the 
portfolio assessment process and the 
results that are generated. 

Educating community groups that 
are interested in and responsible for 
education must parallel the teachers' 
experiences in the classroom. If port- 
folio assessment is to deliver on its 
great potential, educating parents 
and community leaders fiilly about its 
elements and its utility is as important 
as training teachers. ^jflk 
A week-long training institute nil^j^ 
not give a teacher the confidence 
needed to understand a new philoso- 
phy of assessment, let alone make 
meaningful changes in instruction. 
The teacher new to portfolio assess- 
ment will need a network of col- 
leagues who can meet regularly to 
share ideas, materials, frustrations, 
successes, anxieties, and triumphs. 
This exchange needs to go on among 
the teachers in any school where port- 
folio assessment is being adopted. It 
also may mean getting help from out- 
side sources. 

In Vermont, educational planners 
divided the state into 17 networks, 
each with a writing and mathematics 
network leader. These leaders are 
teachers interested and involved in 
the portfolio assessment process. 
They are responsible for contacting 
each school in tlieir area regularly to 
learn what services the school feels 
are necessary to help their teachers 
adopt the portfolio assessment pro- 
cess. A network coordinator trains 
network leaders in using supplemel|||P 
tary materials. The leaders then make 
presentations to schools to help them 
understand the criteria and to help 
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teachers change the emphasis of their 
instruction to meet the criteria. 

Once the teacher understands the 
criteria and has provided students 
with rich problem-sohdng opportuni- 
des, the actual assessment takes place. 
The teacher must be trained to recog- 
nize how well each criterion is being 
met The training includes having 
available anchor or benchmark pieces 
of student work to indicate each level 
of attainment for each criterion. 

There is no substitute for experi- 
ence. A teacher needs to see a range 
of student work to distinguish levels of 
attainment Teachers must check their 
reliability in scoring with other teach- 
ers to be sure they see the standards 
through a common lens. In Vermont, 
this training is done in the networks. 
Each network checks its reliability 
within the network and then checks 
reliability among networks. 

This is an cxdting time in education. 



For too long, what has been taught in 
the classroom has been driven by what 
is easy and quick to assess. Now all those 
involved in education are discussing 
what it is they value in educating and 
how best to assess what thsy value. 

Any changes in assessment, how- 
ever, will call on educators to develop 
a new and richer imderstanding of as- 
sessment techniques — ^and almost 
certainly to make substantive changes 
in instructional methods as well. 
Teachers are trying to come to grips 
with new assessment tools and the im- 
plications they bring for classroom in- 
struction. The success of any alterna- 
tive system will depend on the skills 
the teachers are enabled to de- 
velop—and the support they receive. 



Clare Forseth serves as a statewide 
trainer in portfolio assessment for the 
Vermont Department of Education. 
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PORTFOLIO 

Assessment 



By Susan Black 

If your school is considering portfolio assessment as an 
alternative to traditional tests, take note: The new approach 
has problems of its own 



ALONG WITH BOOK BAGS AND LUNCH BOXES, 
many srudents now tote something new to 
school — portfolios of their work. The use of 
portfolios is becoming increasingly popular in 
U.S. schools as teachers look for alternatives 
to traditional tests to measure student progress. But so 
new is the portfolio concept that there isn't yet much 
research to guide educators in setting up new systems. 
And that should signal a go-slow approach. 

In fact, one recent research repon from the RAND 
Corp., evaluating Vermont's portfolio assessment pro- 
gram, points to some serious problems with portfolios 
that should serve, according to report author Daniel 
Korecz, as a "warning call for people to be a little more 
cautious." 

Portfolios appeal to educators for good reason, 
though. Researchers have long noted that traditional 
tests, standardized or otherwise, have clear limitatioris. 
According to Joan L Herman and S, Golan, for exam- 
ple, the content of the tests too often determines what 
is taught. These researchers (along with many others) 
find that tests ""narrow the curriculum" to basic skills 
rather than higher-order thinking skills. Another re- 
searcher, Lorrie Shepard, reports that even when stu- 
dents do well on tests (often because teachers "teach to 
the test"), it doesn't mean they've learned anything 
valuable. Most likely, Shepard says, what they've 
learned is to take tests well. 

Many teachers, for their pan, maintain there's little 
match between what they teach and what tests mea- 
sure. Teachers who stretch children's minds beyond 
simple memorization and who emphasize group prob- 
lem solving and cooperative learning protest that stan- 



Sunan Black ix an education consultant who lives in Penn Yan, 
S'.Y. 



dardized tests don't reflea their students' real knowl- 
edge and abilities. 

Portfolios are one answer in the search for alterna- 
tive ways to assess students' performance. Portfolios are 
supposed to represent what students know — to show, 
over time and in a variety of ways, the depth, breadth, 
and development of student's abilities, according to re- 
searchers Lorraine Valdez Pierce and J. Michael O'Mal- 
ley. 

But as any teacher who's tried portfolios will anest, 
deciding to use them is the easy pan; it's much harder 
to ensure portfolios accurately record and measure stu- 
dent performance. Some state education departments 
and research labs do offer workshops ' nd booklets on 
portfolio assessment. But teachers usually find it's best 
to experiment and develop their own strategies to fit 
their subjea areas and their classrooms. The work of 
Ohio State University researcher Robert Tiemey and his 
public school teacher colleagues encourages teachers 
along those lines. 

But even under supportive circumstances, the 
change from traditional testing to portfolios isn't always 
easy. Judith Arter, a researcher with the Northwest Re- 
gional Educational Laboratory, says few teachers who 
are excited about the possibilities of using portfolios 
have worked out exactly what rhey mean by portfolios 
or how they should be used. And, says Arter, most 
teachers haven't anticipated or addressed the fallout is- 
sues that can accompany portfolio assessments. 

Using portfolios without a clear plan can lead to mis- 
understandings with parents, administrators, and stu- 
dents. Teachers might also find some tasks bewildering, 
such as sening acceptable standards for student work, 
coordinating assessments with grading requirements, 
and storing archives. If teachers, feel ovejwhelmed with 
the planning details, they might forsake their good in- 
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tentions. And if that happens, student portfolios might 
end up in the circular file. 

The Vermont experience 

In one of the best known portfolio programs, Vermont 
teachers helped the state department of education de- 
sign a statewide system of portfolio assessment as one 
way of evaluating the re- 
sults of a new writing pro- 
gram. The portfolios are 
now used in grades four 
and eight, with grade 11 
to be added soon. A typi- 
cal Vermont fourth- 
graders portfolio contains 
these pieces: (1) a table of 
contents listing the pieces 
the student has selected; 
(2) the best piece of writ- 
ing as chosen by the stu- 
dent: C3) a letter — written 
by the student to the 
teacher and other review- 
ers — aboot the best piece, 
explaining why the stu- 
dent chose it and the pro- 
cess u.sed to produce the 
final draft: (4) a poem, 
short storv'. play, or per- 
sonal namuive: (5) a per- 
sonal response to an 
event, program, or item of 
interest: and (6) a prose 
piece from any subject 
area other than English or 
language arts. 

Vermont, which also 
uses portfolios to evaluate 
students' math progress, is 
a pioneer in implementing 
the approach, beginning 
in 1988. But the state is 
also a pioneer in coming 
face-to-face with the prob- 
lems associated with port- 
folios. The evaluation by 
the R.\.\"D Corp.. released 
in December, showed 
rater reliability — that is. 
the odds that two differ- 
ent teachers would rate a 

portfolio the same way — to be very low. The recom- 
mendations put forth in the report include improving 
training for teachers in how to score portfolios accu- 
rately and making changes in the scoring system itself, 
which the report suggests might be too comple.x. The 
report's author. Daniel Koretz. says Vermont is actively 
looking at instituting some of the changes and that 
RA.N'D will continue its evaluation. 

Koretz. a resident scholar at RAN'D. says those who 
hti^in using portfolios often want to accomplish two 
tilings: Improve what goes on in the classroom, and 
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WHY USE PORTFOLIOS? 



•.-■■4 



Here are some reasons to use portfolios;', as set 
down by English and language aiis teachers in . ■•/, 
California. These reasons are excerpted from Test- 
ing for Learning, by Ruth Mitchell, and adapted from 
Portfolio News, published by Portfolio Assessment ' 
Clearinghouse, c/o San Dieguito Union High School 
Distria, Encinitas, Calif. - ' •;.;;.■'., :•' . 

1. As a teaching tool ' . r.; r"; yiS CT . ., 
— to provide students ownership, motivation; a . . : 

sense of accomplishment, and participation. 
— to involve students in a process of self-evaluation 
— to help students and teachers set goals 
— to build in time for reflection about students' ac- 
complishments 

— to aid in parent conferences 

2. Professional development of teachers 

— to study curriculum and effective teaching 
practices 

— to allow for better staff communication 
— to reduce the paper load 
— to identify school strengths and needs for 

improvement 
— to build a sequence in writing instruction 
3- Assessment 

— to serve as an alternative to standardized testing 
— to serve as a college application and high school 

placement vehicle 
— to replace competency e.xams 
— to serve as a grade or end-of-year culminating 
activity 

— to provide program evaluation 
— to supplement or substitute for state assessment 

tests 
4. Research 

— to examine growth over time and progress in 

students' writing 
— to look at the revision process. 



find a good assessment tool. But. he says, those two 
goals can be at odds with each other. Improving what 
goes on in the classroom often means broadly training 
every teacher in rating student work. Assessing students 
accurately, on the other hand, could well require train- 
ing only a small number of teachers at a time, but train- 
ing them carefully and thoroughly. 

The question that needs to be addressed, Koretz 

says, "is how to compro- 
mise between a powerful 
educational intervention 
and a decent assessment 
program." 

Koretz also says it's im- 
porunt for school districts 
that opt for portfolio as- 
sessment to put in place a 
means for assessing the 
program. "People ought to 
have realistic expectations 
about how quickly (imple- 
menting a portfolio sys- 
tem! can be done and how 
it will come out the first 
time." he says. "A lot of 
people around the country 
have unrealistic expecta- 
tions." 



Making way for portfolios 

Before moving toward the 
use of portfolios, other re- 
searchers agree, teachers 
must first think through 
their reasons for using this 
alternative assessment ap- 
proach. Do they want to 
improve curriculum and 
teaching, or to assess stu- 
dent work — or perhaps 
both? (See the box on this 
page for a list of possible 
purposes for portfolios.) 

Teachers also must de- 
fine exactly what they 
mean by portfolios. Judith 
Aner defines a portfolio as 
"a purposeful collection 
of student work that ex- 

hibits to the student and 

others effort, progress, or 
achievement in a given area or areas." F. Leon Paulson 
and Peari R. Paulson emphasize process over product 
in their definition: "A portfolio is a carefully crafted por- 
trait of what someone knows or can do." Teachers 
might find someone else's definition suitable, or they 
might choose to write their own. 

Then there are the nitty-gritty decisions. What should 
be included in a portfolio? Who should select the con- 
tents? Should a portfolio reflea only a student's best 
work, or should it represent a spectaim of accomplish- 
ments and efforts? Should bulkv items (such as science 
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projects) be considered? What should the school keep 
tor its permanent records? How should teachers com- 
municate students' achievement to parents? How should 
teachers evaluate portfolios? What other kinds of assess- 
ment should be used, if anv? 

.\nsv.ers to some of these questions can be found in 
research. Judith .\rter notes that portfolio contents 
should be chosen according to their purpose. She finds 
that, in general, teachers either require cenain items 
from each student or, with students, choose samples of 
^•ork chat reflect gro^ih and development in a specific 
subiect area. 

F. Leon Paulson and Pearl R. Paulson find portfolios 
should be more than just collections of students work. 
Portt'olios. they say, ought to include students' narra- 
tives about how they produced the contents and about 
^•hat they learned. Students' written reflections about 
their learning might be among the most valuable pieces 
in the portfolios, these researchers say. 

The Paulsons maintain that "students own their port- 
folios," so they — not teachers — should create their col- 
lections and review their selections. They suggest that 
portfolios should tell a student s story, and anything 
that helps tell that story could be included — classroom 
assignments, finished or rough drafts, work students de- 
velop specifically for the portfolio to show their inter- 
ests and abilities, self-reflections, and observations and 
comments by teachers or parents. 

In her report. Sharon Alehouse provides diagrams to 



guide teachers, students, and parents as they choose 
items for students' reading and writing portfolios. (Alt- 
house's study found that teachers and soidents most 
often seleaed ivriting samples for portfolios.) 

California math teacher Pam Knight encourages her 
algebra students to choose from their semester's wonh 
of classwork and homework to construa well-balanced 
portfolios. Knight's soidents are likely to include long- 
term projects, daily notes, journal entries about difficult 
test problems, scale drawings, best and worst tests, and 
homework samples. 

In some remedial programs, students design portfo- 
lios that align with their individual education plans. In 
Pennsylvania's Eastern Lancaster County School District, 
for example, teachers first identify two or three goals 
for each student in Chapter 1 reading and math. (Stu- 
dents may add goals of their own.) Portfolios are likely 
to include written compositions, records of books read, 
examples of drafts and revisions of written work, sam- 
ples of math processes and problems, and original math 
story problems. 

Teachers and researchers also repon successful ven- 
tures using portfolios with high school science and so- 
cial studies students. Missouri teachers use computer 
portfolios to capture their fifth-graders' reading 
progress. In Wyoming, elementary stvidents use laser 
disk technology to record their verbal ability, physical 
accomplishments, anistic achievement, and self-assur- 
ance. Lorraine Valdez Pierce and J. Michael O'.Vlalley 
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describe how elementary and middle school students 
learning English as a second language use portfolios to 
show oral language and reading skills. 

Other concerns 

Time and grades are among the other issues to consider 
before going ahead with portfolios. Managing portfolios 
takes time, that precious classroom commodity. But, re- 
searchers report, teachers who change from traditional 
assessment to portfolio assessment are more likely to 
manage their time without frustration if they change 
teaching styles at the same time. Rather than continu- 
ously assigning and grading workbook lessons, teachers 
should prompt students to learn through writing and 
exchanging ideas. Teachers 
can more efficiendy and ef- 
fectively guide instruction 
through cooperative learn- 
ing groups. And teachers 
should hold conferences 
with their students to rein- 
force and motivate their 
learning and, when neces- 
sary, to reteach prerequisite 
skills. 

Then there's the sticky 
issue of grades. How can 
teachers assign unit grades 
or report card grades (usu- 
ally required by the district 
office, and perhaps by the 
state education depart- 
ment) when they're assess- 
ing students' portfolios for 
effort, progress, and insight 
as well for as specific 
achievement' Some districts 
are experimenting with 

new kinds of report cards — using checklists and narra- 
tives, for example — chat more closely reflect their new 
assessments. 

It's imperative that teachers inform and educate stu- 
dents and parents about new grading systems. Even 
when they're pleased with their portfolios and their 
teachers' comments, some students demand familiar let- 
ter grades — especially when they're accustomed to 
earning A's and B's. Sometimes parents give portfolio 
assessment systems cool receptions because they don't 
understand the new evaluation reports. They might pre- 
fer their children's report cards to look just like the 
ones they used to bring home from school. 

It's imponant to discuss a new portfolio assessment 
system with students and parents. In Sharon Althouse's 
Pennsylvania pilot project, students doubted their par- 
ents would understand the portfolios, even when teach- 
ers enclosed letters explaining the new assessment 
plan. And. Althouse reports, students often changed the 
contents of their ponfolios when they knew their par- 
ents would examine their selections. Though parents 
expressed appreciation and approval of the portfolio 
system, they provided skimpy answers or no answers at 
all to a shon survey about the new method. 
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Finally, other considerations might arise. In high 
schools, students and parents might objea to portfolio 
assessment on the grounds that college admissions of- 
fices require grades and class rankings. And at any 
grade level, serious questions remain about the objec- 
tivity of portfolio assessment. Any program of portfolio 
assessment must address the possibility that assess- 
ments might be biased on the basis of race, sex, or cul- 
tural orientation or overiy generous so as to bolster stu- 
dents' self-esteem. 

A successful start 

If your schcxjl or school distria is considering using port- 
folio assessments, what can you do to help the new ap- 
proach succeed' You can 
f^MBi begin by setting agendas 
for staff planning and staff 
development. The North- 
west Regional Educational 
Laboratory (NWREL) pro- 
poses tackling these topics: 
purpose, curriculum and 
instruction, content, assess- 
ment, management and lo- 
gistics, and staff develop- 
ment. Specific questions to 
discuss, NWEL researchers 
say, include: What are the 
purposes for using portfo- 
lios? How will portfolios re- 
flect the school's curricu- 
lum' How must instruction 
change to suppon portfolio 
assessments? What is ac- 
ceptable in a student pon- 
folio? 'Who owns the port- 
folio, and who chooses the 
contents? What other types 
of student assessment will the school use? How will 
portfolio assessments be coordinated among grade lev- 
els? How will assessments be communicated to 
parents? 

You need to invest time and effort helping teachers 
before they begin using portfolio assessments, but you 
also need to offer support to teachers who might en- 
counter problems <3/rer they've started up their new sys- 
tem. Peer coaching — pairing teachers who are reluaant 
or uneasy about using student portfolios with teachers 
who easily are incorporating the new method in their 
classrooms — might help teachers who want to throw in 
the towel and return to traditional testing alone. Work- 
shops at which teachers examine models and then 
work out their own plans can also help get portfolio as- 
sessments off to a smooth start. 

It's a long and often rocky road to institutionalize 
any innovation in education. Even when teachers are 
eager to accept a new plan, you can bet the process 
will be one of fits and starts. When it comes to devel- 
oping portfolio assessments, you 11 need to encourage 
teachers not to give up when they face difficult issues. 
But you'll also have to remind them — and yourself — to 
go slow with this new approach to assessment. (fl 
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Assessing the Outcomes of 
Computer-Based instruction: 
The Experience of IViaryland 

by DR. GITA Z. WILDER, Research Scientist 
and MARY FOWLES, Senior Examiner 
Educational Testing Service 
Princeton, N.J. 



As computers have been introduced 
more widely into schools, teachers 
and school administrators have 
been called upon increasingly to 
justify the expenditures for hardware and 
software. Typically, school boards and taxpay- 
ers have demanded evidence that computers 
are making a difference, usually in student 
achievement and most often measured by 
scores on standardized tests. 

Such demands take little account of the 
actual conditions that mark the implementa- 
tion of technology in schorls. These include 
the fact that implementation is a process that 
proceeds over a period of several years; that 
computers are used in a wide variety of ways 
by teachers and schools and for a range of 
purposes; and that the widespread use of in- 
structional computing is too new to have been 
supported by a systematic body of research 
about what does and does not "work." More- 
over, many schools and districts have only 
loosely specified objectives for the technolo- 
gies that they adopt. 

This paper describes one effort by several 
school districts to assess the effects of intro- 
ducing computers into schools. The effort 
spanned a three-year period. At the end of the 
three years, all of the participants agreed their 
work had only just begun. 

■ The Sites 

In 1988, the Potomac Edison Company, the 
utilities company that serves three counties in 
western Maryland, initiated a program 
tluough which the firm donated computers 
and educational software to schools in the 
areas it serves. From the outset, one key 
project goal was to demonstrate the potential 
benefits of technology for education. 

Potomac Edison provides hardware (net- 
worked computers, typically), software and 
traiiung; the schools develop a plan for using 
the technology and provide release time for 



teachers to participate in training. 

Staff members Mary Fowles and Gita 
Wilder of Educational Testing Service (ETS) 
worked with Barbara Reeves and Patrida 
Mullinex of the Maryland State Department of 
Education (MSDE), officials of Potomac 
Edison and with members of an Evaluation 
Corrunittee of the larger project to develop a 
plan for evaluating this effort to introduce 
technology into the schools. 

■ The Challenge 

Each district (and sometimes each school 
within a district) developed its own plan for 
using the technology. Moreover, the comput- 
ers were utilized by different districts and /or 
schools for very different instructional pur- 
poses. Thus, a uniform evaluation plan 
seemed out of the question. 

As a result, the first year of the project was 
spent in a series of planning sessioiis that in- 
volved school and district administrators in 
addition to representatives of Potomac Edison 
and MSDE. The group agreed the task of 
evaluating so diffuse a "treatment" was be- 
yond the reach of any conventional program- 
evaluation design. At the same time, it was 
also acknowledged that some form of account- 
ability was called for, and that standardized 
test results, eilthough judged as inappropriate, 
would have to be the default measure should 
the group fail to devise its own approach to 
evaluation. 

There were, moreover, several levels of 
"outcome," each with its own set of questions 
about the effects of computer use on indi- 
vidual staidents, on teachers and dassroonis, 
and on schools. The broadest forms of the 
questions were, how does the introduction of 
computers into schools affect the learning 
process for students, the instructional process 
of teachers, the dassroom process for both 
students and teachers, and the organizational 
processes of schools? 
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■ The Evaluation Process 

The group reviewed (and agreed to incor- 
porate) a set of questioniuures developed by 
the MSDE for use in a statewide evaluation of 
computer use. At the same time, there was a 
strongly felt need to collect data that reflected 
the spediic activities and goals of the Potomac 
Edison project. By the end of the first year, it 
had become clear that the teachers who were 
using the technology were the most appropri- 
ate participants in any effort to document the 
accomplishments of the project. 

The New Assessment Measiu-es Committee 
of the Maryland Education Project first met in 
winter 1990 to devise an overall strategy for 
data collection. Becatise most of the computers 
had been installed in elementary schools, the 
committee was made up mainly of elementary 
school teachers, administrators and computer 
coordinators. 

■ Work Samples as a First Step 

The first meeting of the group was devoted 
to defining the task and to general discussion 
of performcmce assessment as a potentially 
fruitful approach to this new area of student 
progress. Members of the committee agreed, 
at the close of the first meeting, to devote the 
spring semester to collecting samples of their 
students' work that appeared to demonstrate 
the imique contributions of the computer to 
the learning process. Each teacher identified a 
set of computer-based assignments that might 
yield appropriate work samples from his or 
her class. Although general gxiidelines for the 
collection of such samples were provided, the 
process was purposely left somewhat open- 
ended so that particulcirly innovative and 
creative work could be included even if it 
failed to conform to the guidelines. 

In organizing their samples for presenta- 
tion, committee members were instructed to 
provide a context for each sample: 

• the configuration of hcirdware and 
software with which each sample had been 
generated; 

• the class and subject area from which the 
sample came; 

• the assignment that resulted in the work 
sampled; 

• the ability level(s) of the students whose 
work had been sampled; 

• what the work sample demonstrated 
about the student's learning; and 

• the feasibility of this as a model or 

Q prototype assignment for other classes. 
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At the second meeting of the committee, 
participating teachers brought work samples 
and supplied information about the contexts 
in which they had been generated. What was 
immediately apparent was the range of dr- 
cumstances witiiin which the teachers worked 
(from labs of 15 or more computers through 
single computers shared by four teachers), and 
the variety of solutions they had applied to 
their particular circumstances. 

The work samples were a rich soimre of 
iiapiration. They ranged from "books" of 
writings produced by individual first-grade 
students through docxunentation and the out- 
come of a class project to assign addresses to 
all of the houses in a small town that had pre- 
viously lacked a system of addresses. 

The most common application among the 
committee members was word processing; 
numerous examples of student work were 
provided that illtistrated and explained how 
students engaged in the writing process. The 
second most common application was the 
organization and presentation of information 
from diverse sources, often facilitating com- 
plex problem-solving that could not have 
taken place without the computer. A third was 
math drill-cmd-practice as a way to support 
classroom instruction. 

In the lengthy discussion that followed 
presentation and review of the samples, teach- 
ers talked about the ways in which the tech- 
nology had enhanced teaching and learning. 
This discussion revealed that the benefits 
teachers perceived from their own and stu- 
dents' involvement with the computers clus- 
tered into a limited number of categories. 
These categories, called "domains" for pur- 
poses of this study, were further discussed 
cmd refined by the committee in a subsequent 
meetmg. 

■ Identification of Domains 

In their discussion of the effects of the use 
of technology in their classrooms, the teachers 
were instructed to focus on areas in which the 
computer appeared to be making a real differ- 
ence in instruction, rather than offering an 
alternative medium for "usual" classroom 
activities. The distmction is best illustrated by 
using the writing process as an example. 

Many teachers reported leading their stu- 
dents through the writing process. Most 
agreed that the vmique contributions of the 
computer to the writing process revolved 
aroimd the capacity of the computer for mak- 
ing revisions easy and visible, and for creating 
products that increased students' pride in 
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their products and interest in reading the writ- 
ing of others. Becavise of the computer, the 
quality of the students' writing improved, 
more people were interested in reading it, and 
communication became more effective. 

From the general discussion of the domains 
emerged a more specific list on which to focus 
data collection activities of the coming year. 
The domains so identified included: 

• use of the writing process (referring to the 
many ways in which the computer illiimi- 
nates the writing process and makes it 
easier for students to write and become 
both proficient and prolific in writing); 

• production and performance for purposes 
of communication; 

• integration of reading and writing; 

• gathering, integrating, analyzing and 
using information; 

• solving complex problems; 

• adapting instruction to the needs of 
individual students; 

• motivation (to engage in leanung); 

• efficiency, productivity and quality of the 
teaching/learning process; 

• application /genenilization (referring to 
spontaneous transfer of learning from one 
area to another on the part of students); and 

• comfort and competence with technology. 

These domains were not mutually exclu- 
sive, nor were they intended to be. Their func- 
tion was to guide collection of work samples 
that would offer evidence of the domains in 
question. Any given sample might offer evi- 
dence of several domains at once. 

■ Framework 

The group then devised a plan for collect- 
ing work samples more systematically during 
the 1991 school year. Each committee member 
selected specific domains in which to concen- 
trate data collection activities, identified one 
or more work samples collected as evidence of 
learning in the designated domain(s), and 
developed a unique plan of data collection. 

From the work samples, a framework for 
assessing student progress had to be aeated. 
Following a (school) year of data collection, 
the commitise re-assembled to consider the 
work samples and re-consider the domains. 
Although some domains proved more fruitful 
than others in the ease with which appropriate 
work samples could be provided, the range 
and variety of samples was most impressive. 

Working in small groups, each of which 
focused on a sub-set of the domains, commit- 
tee members reviewed work samples from the 



perspective of the evidence each provided 
about the level of competence demonstrated 
by students (and often teachers) in the domain 
in question. 

The results of this process were a set of 
scaled indicators of performance within spe- 
cific domains that might be used to assess 
progress in the application and use of technol- 
ogy. Teacher and student progress were de- 
scribed separately. And, by agreement of the 
committee, each set of indicators included a 
"top performance" rating that transcends the 
highest level demonstrated in any of the work 
samples. This rating reflects the committee's 
recognition that new technology and new 
applications thereof proceed at a rate that 
cannot possibly be anticipated. The tixmscen- 
dent rating allows for the incorporation of 
unforeseen, creative uses of the technology. 

The ratings exist as a set of matiixes that 
describe and provide examples of graded 
performance using technology within a speci- 
fied set of domauis. The committee agreed 
that development of the matrixes is only the 
first step in an active process that will involve 
others outside of the committee. Committee 
members plan to apply the matrixes in their 
own dassroonis, schools and disbicts. Staff 
members from the MSDE are disseminating 
the matrixes and inviting reaction from a 
wider set of schools and districts within the 
state. And readers are encovtraged to contact 
the authors, or the Maryland Education 
Project at the address below, to obtain copies 
for their own use. 

Undoubtedly, the matrixes will be revised 
with time and use. Yet all agreed that they, 
and the process that produced them, were 
invaluable alternatives to the use of standard- 
ized test results to assess the effects of imple- 
menting technology in schools. ■ 

For more informatioii or copies of the matrixes: 
Educatioi»al Testing Service 
Attn; G. Wilder or M. Fowles 
Rosedale Road 
Princeton, NJ 08541-0001 

Gita Wilder is a rtsearck scientist in the Division of Education 
Policy Research at Educational Testing Service and director of a 
program of research on the organizational context for change. 
The focus of that research is school change engendered by tht 
implementation of technology. 

Mary Fowles is a senior examiner in the Humanities Group. 
Test Development Division, for School and Higher Education 
Programs at Educational Testing Service. Her work includes 
the development of assessment measures in literature and 
writing. Most recently, she has collaborated with seoeral school 
systems and states in the development of portfolio asstssmtnts 
for use ma variety of subject areas. 
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Why Standards May Not 
Improve Schools 



Elliot W. Eisner 
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If we value student work 
that displays ingenuity and 
complexity, we must look 
beyond "standards" 
for evidence of achievement. 



Few ideas are more central to the 
educational refomi movement 
currently underway in America 
than that of standards. Virtually 
everyone thinks we need them 
and that efforts to improve our schools 
will certainly fail unless standards can 
be develojjed and made public. Stan- 
dards will provide both targets and 
incentives. Students will know what to 
aim for; teachers will understand how- 
effective they have been; and. perhaps 
most important, the public will know 
how well schools are succeeding. 

Amidst the enthusiasm, neverthe- 
less, an undercurrent suggests that 
standards may not be the answer and 
that the concept itself may impede the 
realization of the vision of education 
that many educators hold. 

At the outset, we should acknowl- 
edge that the term "'standards" has 
multiple meanings. Standards can 
refer to those targets at which one 
aims. Standards can serve as icons 
against which student performance is 
compared. The term "standard" can 
also refer to something that is 
common or typical. Standards are 
sometimes used in relation to a rite of 
passage that provides access to future 
opportunities. For example. "He has 
met the standards we have set and is 
now able to practice medicine." 

In addition to these conceptions, 
standards, as Dewey pointed out in-4rt 
as Experience, are units of measure- 
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ment.' We have a standard unit of 
measurement we call ounces, another 
we call pounds, and still others called 
inches, feet, miles, kilometers, and 
statute miles. All of these standards 
are socially defined units that quantify 
qualities; What we experience as 
height, we can transform into feet and 
inches. What we experience as heat, 
we can transform into temperature. 

There is, of course, a fundamental 
difference between the experience of 
heat or height and their description 
thro!jigh standards we call centigrade 
or inches. Knowing heat as an experi- 
ence is not like knowing heat as 
temperature. Thus, one important 

We seek work that 
displays ingenuity, 
complexity, and 
the student's 
personal signature. 



feature of a standard as a unit of 
measurement is that it functions as a 
symbol that possesses none of the 
qualities of what it has measured. 

Another feature of standards, and 
this feature is crucial from an educa- 
tional perspective, is that, as a unit of 
measurement, a standard is a vehicle 
for describing, rather than appraising, 
a set of qualities. When we apply stan- 
dards in Dewey's terms, we get 
answers to questions that pertain to 
matters of amount. Yet what we want 
regarding the outcomes of our 
teaching are not simply matters of 
amount but matters of goodness. We 



want to achieve, or help students 
achieve, what we or they value. 

Given this latter conception, stan- 
dards by themselves will never be 
adequate for detemiining whether 
what a student has done or under- 
stands is of value. To determine 
matters of value, we need something 
more. 

Criteria, Not Standards 

The something more we need are 
criteria. Criteria facilitate the search 
for qualities we value within an essay, 
■a scientific experiment, a painting, a 
work of history, and the like. These 
works, Dewey argues and I concur, are 
not susceptible to measurement by 
standards, although they are amenable 
to appraisal by criteria. 

Having said that, it should be 
acknowledged that once a standard is 
assigned a value, it can be applied to 
those forms of performance or prod- 
ucts for which there is a fixed correct 
answer. With a standard, we simply 
lay down the appropriate unit and 
count the correctness or incorrectness 
of the responses. We can apply stan- 
dards to spelling and to arithmetic. We 
can apply standards to most forms of 
punctuation and to the determination 
of standard grammatical usage. While 
the application of standards to such 
topics is useful and efficient, standards 
do not represent the most important 
ends we seek in education. What we 
value in education is not simply 
teaching children to replicate known 
answers or to mimic conventional 
forms. We seek work that displays 
ingenuity, complexity, and the 
student's personal signature. In short, 
we seek work that displays the 
student's intelligent judgment. Work 
of this kind requires that we also exer- 
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cise judgment in appraising its value. 

This judgment depends, I am 
arguing, on the availability of criteria. 
The application of criteria requires the 
exercise of judgment, the ability to 
provide reasons for the judgments we 
make, and an understanding of which 
criteria are relevant to the genre of the 
work. Applying criteria is a much 
more complicated and intellectual 
enterprise than applying a standard. 
Hence, when we talk about using stan- 
dards as a lever for educational 
reform, we grossly oversimplify what 
is required. 

The problem of assessment, 
however, is considerably more 
complicated than the distinction 
between standards and criteria that I 
have just drawn. Within the context of 
schooling, teachers do — and indeed 
they ought to — take into account not 
only the qualities of the student's work 
but other considerations that pertain to 
the individual student. Experienced 
and skilled teachers know that when 
they appraise a student's work, they 
need to consider where that student 
staned, the amount of practice and 
effort expended, the student's age and 
developmental level, and the extent to 
which his or her current work displays 
progress. Although such considera- 
tions are not particularly relevant for 
appraising the work of professionals, 
they are relevant for appraising the 
educational development of the 
student. 

But even if we were to consider 
only the student's work, we would still 
have the problem of determining how 
standards — even when they are rele- 
vant to the work — should be derived. 
Should standards be derived from the 
performance of the particular popula- 
tion of students from which a smdent 
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comes or from the school district as a 
whole, the state, the region of the 
country, or the nation? Should stan- 
dards be formulated from the perfor- 
mance of students in suburbs or inner 
cities? From those in tiie 50th 
percentile or in the 75th? Is it fair to 
have the same standards for all 
students when we know that some 
students come to school without 
breakfast while others are driven to 
school in expensive automobiles? 
Should we employ the same standards 
in schools that barely have enough 
resources to open their doors while 
others are the educational equivalent 
of Neiman Marcus? Just what is 
appropriate, and just what is fair? 

Furthermore, what do we do with 
the information after the standards 
have been applied? Will we know 
what needs to be done by examining 
the performance of students in relation 
to fixed standards? 

It is characteristic of our culture, 
and particularly our attitude towards 
education, to seek some magic wand, 
some golden lever, that we can 
employ to make things right I do not 
argue here that our schools are in good 
shape. Some schools are wonderful, 
many schools are dreadful, and most 
schools, from my perspective, need to 
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generate a much greater sense of intel- 
lectual vitality and challenge than they 
now possess. I cannot help but wonder 
whether this emphasis on standards is 
likely to move schools in the direction 
that I value. I do not value schools that 
regard children as an army marching 
toward fixed and uniform goals. Stan- 
dardization is already too pervasive in 
our culture. We need to celebrate 
diversity and to cultivate the idiosyn- 
cratic aptitudes our students possess. 
Certainly, an array of common 
learning is appropriate for almost all 
students in our schools, but the preoc- 
cupation with uniform standards, 
common national goals, curriculums, 
achievement tests, and report cards 
rings in a theme that gives me pause. I 
believe we would do far better to pay 
more attention to the quality of our 
workplace and to the character of our 
teaching than to display such preoccu- 
pations with standards. If we can 
create schools that excite both teachers 
and students and provide the condi- 
tions that improve the quality of 
teaching, we will do much to create 
schools that genuinely educate. 

The current emphasis on standards 
will provide no panacea in education. 
Paying close attention to how we 
teach and building institutions that 
make it possible for teachers to 
continue to grow as professionals may 
be much more effective educationally 
than trying to determine through stan- 
dard means whether or not our 
students measure up. ■ 

'J. Dewey, (1934), An as Experience, 
(New York: Mimon Balch and Company). 



Elliot W. Eisner is Professor of Education 
and Aft, School of Education, Stanford 
University, Stanford, CA 94305-3096. 
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In looking closely at what is happening in Rhode Island, Mr. Maeroff notes, 
it is possible to get a preview of the satisfactions and frustrations that may spread 
as the movement toward large-scale alternative assessment mushrooms. 



By Gene I. Maeroff 



WILLIAM, a tiny third-grader in a red 
sweatshirt, was sitting in an adult- 
sized chair, balancing on the edge of 
the seat as he listened intently to the 
woman across the table telling him that she was go- 
ing to ask him about one of the several books tiiat 
he had read recently. 'If you know the questions Fm 
going to ask, it may help you decide which book you 
want to discuss," she said, going on to Ust the sub- 
jects of the questions. 1 will want to know about the 

GENE I. MAEROFF is a senior fellow at the Carnegie 
Foundation for the Advancement of Teaching, Prince- 
ton, N.J. , and author of The School-Smart Parent (Henry 
Holt, 1990). 



main part of the story, the main characters, the set- . 
ting, the ending, and the major conflict or problem." 

And so it was that William, aware in advance of 
what he would be asked, decided to answer ques- 
tions about a biography of Marco Polo &at he had 
read not long before. What ensued between child and 
adult resembled an informal conversation more than 
the assessment that it actually was. They go quite 
a bit by camel through the desert," William said of 
the setting as he answered the questions, one by one. 
"He lived in Venice, and they were going to trade 
some things from Italy for some better things. A war 
broke out, and so they decided to go to Qiina and 
stayed there." 





For all Us attractiveness, alternative assessment 
is fraught with complications and difficulties. 



s, 



"Does it end in China?" William was asked. 

"No," he answered. "After 17 years he decided to go home 
and left on a fleet of Qiinese junks. When Marco Polo got 
back to Italy, no one believed iidm, so they put him in prison. 
He wrote a book about it, and years later Christopher Colum- 
bus read it." 

This encounter was an assessment of William's reading, and 
the only paper and pencil in evidence were in the hands of 
the adult who was questioning him and taking notes on his 
responses. None of the questions had 
multiple-choice or true-or-falsc answers, 
and nothing about this reading examina- 
tion was norm-referenced or nationally 
standardized. It was an alternative assess- 
ment, which included the individual in- 
terview about the book on Marco Polo 
and a review of a poitfolio containing 
samples of William's work. Later, with 
another assessor, William would be as- 
sessed on other tasks, participating in a 
small group in which he would be asked 
to read and discuss a different story and 
to write about it. 

Alternative assessments of this sort 
typify a movement that is capturing the 
imagination of educators across the coun- 
try. Good teachers have historically used 
such methods to monitor the progress of 
their students, but now these approaches 
are being extended beyond individual classrooms to pose a 
challenge to traditional ways of mass testing. However, large- 
scale alternative assessment is still talked about more than it 
Is used, and William's experience in his school in South Kings- 
town, Rhode Island, represents one of the early efforts by a 
state to develop a system of alternative assessments in its 
elementary schools. 

Other states - noubly California, Connecticut, Kentucky, 
and Vermont - as well as some school systems and individu- 
al schools are at various stages in this gentle upheaval that 
proponents hope will alter the way Americans think about 
evaluating schoolwork. Altogether, 40 states are planning some 
form of alternative assessment at the state level, with writing 
samples as the most common alternative.' 

Rhode Island's pilot project, which began in 1989, is still 
inchoate as a small number of third-grade teachers in four 
school districts around the state collaborate with officials from 
the state education department and a researcher from the 
Educational Testing Service (ETS). Together they are strug- 
gling to devise methods of assessing students that can provide 
useful information while avoiding the shortcomings associat- 
ed with norm-referenced tests. In looking closely at what is 
happening in Rhode Island, it is possible to get a preview of 
the satisfactions and frustrations that may spread as the move- 
ment mushrooms. 



peed and low 
cost were the silver 
bullets that enabled 
norm-referenced tests 
to carry the day. 



Those pursuing change in Rhode Island are continually ad- 
ministering and revising the pilot assessments, striving to cre- 
ate a framework for instruments that evenmally might be used 
in schools throughout the state at various grade levels. They 
hope to foige a synergy of instrucdon and assessment in which 
each complements the other to raise learning to new levels. 
This approach allows a child to know in advance - as Wil- 
liam did — what will be asked. The goal is not to spring sur- 
prises on smdents and catch them unawares; unlike the usual 
mode of testing, this is no "gotcha" game. 
The students, under the tutelage of their 
teachers, are trained to provide evidence 
of their own learning. 

The original charge to the pilot group 
in Rhode Island was to try to determine 
how the sute's "Outcomes for Third- 
Graders" might be measured through the 
use of portfolios. This goal, which fits 
comfortably within the nation's alterna- 
tive assessment movement, has extend- 
ed beyond portfolios to include other pos- 
sibilities, such as performance tasks and 
group interviews, and it embraces out- 
comes in reading, writing, speaking, lis- 
tening, and mathematics. 

Schools presumably are doing their job 
if smdents learn something that is deemed 
worth knowing. This kind of assessment 
does not drive the curriculum; it grows 
out of the curriculum and is part and parcel of the curricu- 
lum. Such a philosophy is widely accepted at the elementary 
and secondary levels in the performing and sudio arts, in ath- 
letics, and even in vocational education. But it is not readily 
accepted in formal academics. Think, for instance, how long 
it was before test-makers asked students to produce essays in 
order to demonstrate their writing skills instead of giving them 
tests with questions about writing. 

A young pianist who is asked to master Beethoven's "Fur 
Elise" becomes proficient by practicing the piece, knowing 
all the while that his examination will consist of playing it. 
A nonswimmer who is told that the measure of her ability to 
swim will be the completion of one lap of the pool endeavors 
to swim with full knowledge of the form the assessment will 
take. Being able to play Beethoven or to swim a lap of 
the pool is presumably indicative of the ability to play some 
other composition of equal difficulty or to swim a similar dis- 
tance on another occasion. 



Y! 



'ET, FOR aU its attractiveness, alternative assessment 
is fraught with complications and difficulties, not un- 
like having to endure life with a teenager as the price 
for the joys of parenthood. If they are to be used 
widely or even as supplements to nationally standardized. 
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norm-referenced tests, these new assessments will have to be 
done more quickly, more efBciently, and less expensively than 
at first seems possible. 

As experimenters in Rhode Island and elsewhere are dis- 
covering, the quest for alternative forms of mass testing could 
remain as elusive as Don Quixote's dream. The schools seem 
unable even to broker the logical marriage of alternative as- 
sessment and technology; they treat the two as if they dwelt 
on separate planets. For example, in elementary and second- 
ary education, how often do we hear of alternative assessment 
that involves the manipulation of computer models or simula- 
tions? 

Furthermore, there must be standardization of some sort, 
as Rhode Island hopes to achieve. Otherwise, there is no way 
to put the fmdings of an assessment in context. Even in Ken- 
mcky, which by 1995 is to have the first statewide assessment 
system that is completely based on performance, there is an- 
guish over how to meet the state board's mandate that there 
be a way of comparing Kentucky smdents with those in other 
states. 

Speed and low cost were the silver bullets that enabled the 
norm-referenced test — with its multiple-choice responses — 
to conquer the world of education and hold it in thrall. Mean- 
while, alternative assessment, which is not so new an idea as 
some people think, tends to be a time-consuming, labor- 
intensive, imprecise exercise in which the expense mounts as 
nuances are weighed and scoring is done by humans, without 
even the benefit of grids that fit conveniently over answer 
sheets studded with blacked-in boxes. 

On the other hand, there is the potential for teasing con- 
siderable consensus from an approach that seems at first to 
be hopelessly subjective. Consider the competition in such 
sports as diving and figure skating, in which experts have de- 
veloped scoring criteria that are so extensively accepted around 
the world that judges find a re- 
markable level of concurrence. 
Some of the best work in assess- 
ing writing samples mirrors this 
sort of agreement on relevant 
criteria. 

While it may be possible to be 
systematic about aitemative as- 
sessment, there are ultimately no 
quick and easy ways to rate large 
numbers of performance-based 
tasks or portfolios or interviews 
or exhibits or even essays. Amer- 
ican proponents of aitemative as- 
sessment like to cite the example 
of England as a country that has 
used aitemative assessments to 
the exclusion of norm-referenced 
and multiple-choice tests. That is 
generally true, but until recently 
it has mostly meant using essays 
for external assessment. Only 
lately has England made a large- 
scale anempt to develop for ex- 
ternal assessment such nonwriting 



tasks as science experiments. Meanwhile, portfolios contain- 

ing work done over time have been and will continue to be|fl|| 
used in England for assessment within the classroom, tiiough 
not for outside comparisons of students or for purposes of ac- 
countability. 

Measurement experts from around die U.S. who gathered 
in California in March 1991 under die auspices of die federal 
government's Center for Research on Evaluation, Standards, 
and Student Testing commiserated over the difficulty of meet- 
ing die rapidly escalating demands of policy makers for alter- 
native assessment tools. They noted die time that it will take 
to establish validity and reliability, the need to train teachers 
in aitemative assessment procedures, and the amount of time 
needed to administer the examinations.^ 

Nonedieless, the pioneers in Rhode Island and odier locales 
— like diose who drove their Conestoga wagons into unfamiliar 
territory - steel themselves and press die joumey forward, 
intent on building an assessment system that is embedded in 
instruction. They want an approach that attests to a student's 
progress over time, something diat figuratively resembles a 
semester-length videotape radier dian a Polaroid snapshot that, 
like a one-shot test, captures only a single moment of a child's 
learning. 

The cynosure for each of five subject areas in Rhode Island 
is a set of literacy outcomes for third-graders. One aim is to 
assess outcomes in reading, writing, speaking, listening, and 
mathematics — both as individual subjects and in ways that 
integrate diem. The intention is diat teachers will then teach 
in this manner. 

In tills approach, assessment drives instruction, and instruc- 
tion drives assessment, much the way the front and rear axles 
impel one another in a vehicle with four-wheel drive. In es- 
sence, the assessment task is part of die instruction. This no- 
tion may be revolutionary for elementary and secondary edu- 
cation, but in medical schools, 
where clinical education is often 
evaluated on die basis of perform- 
ance tasks diat may have an in- 
structional component even dur- 
ing die examination, it is accept- 
ed practice.' When assessment is 
sheared off from instruction, as 
is customary in the commercial- 
ly produced examinations given 
in the public schools, die findings 
may not tell a great deal about 
what students have learned from 
their classroom experience. 

One must wonder, therefore, 
about the movement for 9 national 
examination system diat is hurry- 
ing forward on several fronts. A 
great deal is heard about various 
plans to create tests diat would be 
given to students at, say, fourth, 
eighdi, and 12di grades. Yet diesel 
discussions are curiously reticent^ 
about die curriculum on which 
5 2 students are to be examined. Pre- 
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sumably, these tests will dictate the curriculum, a policy that 
U anathema to proponents of alternative assessment. The con- 
cern about widespread low achievement that is fueling the 
movement lor a national test is understandable. But wouldn't 
it be more straightforward to decide what ought to be taught 
and to teach it, instead of first administering tests and hoping 
that they push the curriculum in the right direcdon? 

Perhaps circuitous routes are chosen because the challenge 
of uiufying teaching and assessment is so daunting. Assess- 
ment that is authentically embedded in instruction is not easy 
to fashion. This is no Operation Desert Storm. There are no 
sman bombs to wipe out specific problems, nor any flanking 
maneuvers to avoid the tedious process of repeatedly refining 
the assessment instruments. There is just the slow, mundane 
reconnoitcring to find and frame the best ways of eliciting the 
most useful informadon. One can only hope that alternative 
assessment is not rushed onto the battlefield of tesdng so hastily 
as to produce in its unperfected form friendly fire that harms 
the very children who are supposed to be the beneficiaries. 

And the work doesn't stop with declaring, for example, that 
students will submit portfolios. What should be in the portfo- 
lios? What should students be asked about the contents of their 
portfolios? How can some element of standardization be lent 
to the process so that one student's portfolio may be compared 
with another's? Putting less emphasis on comparisons is fine, 
but at some point a child and his parents have a right to know 
whether the child's progress is reasonable for his or her age 
and experience. 



THERE IS an inclination among proponents to regard 
alternative assessment as suitable for all purposes - 
diagnosis, selection, and accountability at all levels. 
It may be that an alternative assessment that is a 
marvelous indicator of an individ- 
ual child's academic progress will 
prove fairly useless for other pur- 
poses. Americans may have to de- 
cide whether comparisons are what 
they seek in alternative assessment 
or whether they prefer to use the 
approach for other, more individu- 
alized purposes. Incidentally, the 
very idea of comparing the prog- 
ress of young children is seldom 
embraced in Europe, where the 
practice is generally nof to test stu- 
dents in the primary years. But 
America is different. 

Thus, one crisp New England 
morning at South Road Elemen- 
tary School, little William and 
some of his classmates played 
guinea pigs for the assessment de- 
veloper, who had situated them- 
selves at three separate sites in the 
school. It was part of a continu- 
ing process of refmement that has 
continued through this fall. 



Grant Wiggins reminds us that the root of the word assess- 
ment means to "sit with" a learner and seek to be sure that 
a smdent's responses really mean what tiiey seem to mean. 
He adds: 'Does a conea answer mask thoughtiess recall? Does 
a wrong answer obscure thoughtful understanding? We can 
know for sure by asking fiirther questions, by seeking explana- 
tion or substantiation, by requesting a self-assessment, or by 
soliciting the smdent's response to die assessment."* 

Next door to the conference room in which William met 
with his assessor, Mary Fowles of ETS, four odier third- 
graders wearing name tags sat at a table with Susan Skawin- 
ski of the state department of education. She was idso carry- 
ing out a pilot assessment, talking about a story that they had 
read and were going to write about. The idea at this point was 
not so much to assess the children's learning as to assess the 
assessment itself. Eventually, classroom teachers are to be the 
assessors of their own students. And when diis happens, they 
will need a fiiily developed instrument. 

This was a group interview in which Skawinski would as- 
sess each student's understanding of literature by having him 
or her discuss and write about what had been read. Tm going 
to talk to you about something you've done, and Fm going to 
talk to you about reading," Skawinski said to die three girls 
and one boy who were gathered with her around the jpor 
mica-topped table. 

"In this story we have some very important characters," she 
said, proceeding to list and discuss the characters in "The 
Rooster Who Understood Japanese," a tale about a Japanese- 
American family that owned a rooster whose crowing woke 
a neighbor each morning. Their home was on the outskirts 
of a city. Skawinski handed out copies of a "story map" to 
the youngsters and, following its scheme, mentioned the 
characters, the setting, the problem, the attempts to solve the 
problem - all of which had been briefly summarized on the 

story map. She took notes on a 
pad on her lap as the students - 
sometimes readily, sometimes re- 
luctantiy - entered the conversa- 
tion. 

The box labeled "solution" was 
blank on the story map, and die 
children were asked to write in the 
details of what had been done to 
deal with the crowii^ of die roost- 
er, named Mr. Lincoln. The stu- 
dents were permitted to go back 
and look at tlie stoiy if diey wished. 
This, it wmed out, was a warm- 
up for the real task: the writing 
assessment. Skawinski asked the 
children to diink about solutions 
in addition to die one offered by 
the author and prodded diem to 
discuss dieir ideas widi die group. 

"Tie a rope around his beak." 
suggested Rebecca,- causing Kev- 
in, sitting next to her, to look 
crestfallen. He said that he had 
had the same idea. 
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"Send him to rooster obedience school," suggested Alicia. 

And so it went as possible solutions to the conflict were pro- 
posed and discussed. Tick a solution you tiunk makes the most 
sense and write about it,' Skawinski said. It can be your own 
or someone else's that we talked about." In addition, the chil- 
dren were asked to address two other questions about the story. 
The following excerpts are part of what each third-grader 
wrote: 

'keep mr. licon in the dark because the moring he crows 
in but not at night" 

"I would put Mr. Lincoln in the dark so he would think it 
was still night and wouldn't crow." 

"^k to him in japaneas and then he will be quiete" 

"Put him in a room when it's dark to make him feel that it's 
still dark out side." 

It took so long to complete the discussion and the writing 
that, as she collected their papers, Skawinski told the children 
that there would not be time for them to talk about their fa- 
vorite books, which they had been asked to bring with them 
to the assessment. "It was a laboriously slow process," she 
said afterward. "I needed more time." Such complaints are fre- 
quent in the pOoting of alternative assessments because a cen- 
tral problem is figuring out how to accomplish the assessment 
in a manageable time period. 

Funhermore, in this case, the assessor would stUl have 
to devote time to sorting out the notes she took during the as- 
sessment. "You can't actually mark the sheets in front of 
the children," Skawinski remarked after the students had 
left. 

She noted discreetly on the pad that had been perched on 
her lap throughout the interview the extent to which each child 
seemed to bring background knowledge to the story — ideas 
about living arrangements in a city, an awareness that roosters 
crow early in the morning, and so on. She noted their familiar- 
ity with a story map as a way of sening out the structure of 
the story. She also observed how long it took each child to 
read the story silently and what responses each one had to ques- 
tions about the story - who had original ideas, who followed 
the ideas of others, how well individual 
students could go back and cite informa- 
tion from the story to justify what they 
were saying. 

Mary Fowles of ETS expects that, as 
the assessment is improved, there will be 
less note-taking by the assessor. In the 
pilot assessments, anention has been paid 
to deciding what information is most im- 
ponant to gather about a student's reac- 
tions and what kinds of evaluation forms 
lend themselves to recording this infor- 
mation. Theoretically, it is possible for 
the evaluation form to be anything from 
a check-off list to a sheet on which de- 
tailed quotes from a discussion with a stu- 
dent are recorded. 

Assessing a ponfolio might seem to be 
a more straightforward job than assess- 
ing students' performance in a group in- 



terview, but the challenge is siniply of a different sort. This 
was illustrated in the trial assessment of mathematics portfolios 
that Mary Ann Snider, a testing and evaluation specialist with 
tfie state education department in Rhode Island, was conduct- 
ing with students at a desk in the corridor outside their class- 
room. 

"Your teacher tells me you've been doing several things in 
math that you might want to share with me," Snider said to 
the first of the students, Nigel, as she began reviewing his math 
portfolio with him after a bit of casual chatter. Nigel was eager 
to talk and spoke enthusiasncally about his work in conjunc- 
tion with a medieval theme, used concurrently by many 
teachers in his school as a means of teaching various subjects 
in an interdisciplinary fashion. There had also been a school- 
wide medieval festival fot~which the classes had gathered en 
masse.. 

Nigel showed Snider a pencil drawing he had made of in- 
tricately linked chain mail, and they discussed what he had 
learned about shapes by making the drawing. "What else did 
you do with math in the medieval project?" Snider eventual- 
ly asked. The emphasis in this part of the assessment wa« on 
finding ways in which math had been integrated Into other 
subjects, reflecting the goal of bringing about the interdis- 
ciplinary teaching that the assessment is intended to rein- . 
force. 

Nigel displayed a word puzzle that he had devised, using 
vocabulary words associated with the medieval period, and 
a separate math word problem that he had wrinen on a medi- 
eval theme. But the math problem he had created was so con- 
voluted that even he could not solve it. Then he showed some 
work that he had done on estimation. It called for counting 
the number of bricks in one portion of a castle wall and es- 
timating the number in the entire wall. Snider initiated a dis- 
cussion of how the drawing of the castle might be used to study 
shapes and asked Nigel to identify some of the shapes. 

Moving beyond the contents of itie portfolio and following 
a wrinen script so that the assessment would be systematic. 
Snider posed a series of questions, some of which were more 
successful than others in eliciting responses that could help 
in assessing Nigel's understanding of 
math: "What has been your fevorite thing 
to do in math this year?" "When you learn 
something new in math, when t'o you 
know that you understand it?" "Ha> e any 
of the new things you've learned in math 
helped you outside of school?" 

Finally, they reached the last pan of 
the assessment, the task that Nigel would 
be asked to perform. A plastic bucket 
filled to the brim with small, square, 
colored tiles was put on the desk be- 
tween them, and Snider held up a bag. 

"I have something in this brown paper 
bag, and I have a riddle so you can fig- 
ure out what is in it," she said. "I will 
give you some clues, and you can use 
these tiles to figure it out Here's the 
rr ,1 first clue: there are fewer than iO tiles 

^ in the bag. What does that mean?" 
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It is easier to propose outcomes than it is to 
set the criteria and establish performance levels. 
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"Less than 10." 

"Docs that mean I could have 10 of them in the bag?" Snider 
asked. 

"Yes . . . ," Nigel answered hesitantly. Then, upon a mo- 
ment's reflection, he changed his answer. 

The goal was for Nigel to take from the bucket and put onto 
the desk tiles of the same number and color as the riddle indi- 
cated were in the bag. He took nine tiles of various colors 
from the bucket and spread them on the desk, now fairly cer- 
tain that there could not be more than 
nine in the bag. 

"Clue 2: there are two colors in the 
bag." And so it went, Nigel having to 
modify the number or color of the tiles 
in fixint of him in response to a series of 
clues. When he thought he had solved the 
problem, Snider gave him the bag so that 
he could take out the tiles and see if the 
number and color matched those he had 
assembled on the desk. 

"Now," Snider said, "Fd like you to 
write a riddle for me using the tiles, and 
I will try to solve it." 

This was not a wholly satisfactory as- 
sessment, according to Snider, who said 
afterward that she wished Nigel had ac- 
cumulated more math work in his pon- 
folio to reflect studies pursued more re- 
centiy than the medieval project, complet- 
ed some six weeks earlier. Furthermore, she worried that the 
warm-up period had not been adequate and that Nigel was not 
sufficiently at ease during the assessment. "I saw emerging 
evidence of understanding," she said. "He is logical but not 
confident. He doesn't have enough math vocabulary, and he 
has trouble talking about the work he did. He is not comfon- 
able explaining. He remembered what estimation was, but he 
could not say enough about it to show that he understood it." 



ment. When Mark Twain Elementary School in Littleton. 
Colorado, began moving into an assessment program that 
would be based on the actual performances of students, a ma- 
jor part of the effort was devoted to developing criteria for 
scoring the performances. The criteria evolved through at least 
10 stages, according to Monte Moses, the school's principal. 
"Every time we gave the assessment, we saw some student 
doing something we couldn't account for on the scoring rubric," 
Moses said. 

One part of an assignment at Mark 
Twain, for example, calls for fifth- 
graders to submit a written research re- 
port. It is scored on a descending scale 
of 5 to 1. A report earns a 5 when it 



tudents ought 
to spend time think- 
ing about questions 

like those to be 
asked by assessors. 



clearly describes the question studied 
and pi^vides strong reasons for its im- 
portance. Conclusions are clearly stat- 
ed in a thoughtful manner. A variety 
of facts, details, and examples are giv- 
en to answer the question and provide 
support for the answer. The writing is 
engaging, organized, fluid, and very 
readable. Sentence structure is varied, 
and grammar, mechanics, and spelling 
are consistendy correct. Sources of in- 
formation are noted and cited in an ap- 
propriate way. 



By comparison, a 3 is awarded to a written report when the 
student 

briefly describes the question and has wrinen conclusions. An 
answer is stated with a small amount of supporting informa- 
tion. The writing has a basic organization although it is not 
always clear and is sometimes difficult to follow. Sentence 
structure and mechanics are generally correct with some weak- 
nesses and errore. Reference: are mentioned, but without ade- 
quate detail.' 



SO mS that Snider and the others who are piloting 
Rhode Island's assessment arc canning out their work 
in selected schools in East Providence, Glocester. 
South Kingstown, and Newport. In each of the four 
panicipating districts, a lead teacher acts as a liaison between 
the project and other third-grade teachers in the district. One 
aim of the project is to develop a format for reporting indica- 
tors of a smdent's progress toward reaching the state's pro- 
posed literacy outcomes. Most experts believe that the reports 
should be more descriptive than just a row of numbers. But 
the repons will include numbers, and the experts are striving 
to detemune what those numbers should represent. 

This attempt to add meaning to the evaluative numbers can 
be appreciated by examining what happened in a place that 
preceded Rhode Island in the shift toward alternative assess- 



The bottom line is that it is easier to propose outcomes than 
it is to set the criteria and establish the performance levels 
that are represented by various achievements. Moreover, if 
students themselves are to take responsibility for their own 
work, the criteria must be spelled out ways that are under- 
standable to children. Then the smdents can go about learn- 
ing how to do what is expected of them. In general, it is desir- 
able that a student spend time thinking about questions of the 
type that will be asked on the assessment; those questions 
should promote and direct the child's learning. 

In Rhode Island, some individuals administering the pilot 
assessments wondered after one of the pilot sessions how use- 
ful it was, for instance, to ask a smdent, "When you learn 
something new in math, when do you know that you under- 
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Despite England's considerable experience with alternative 
assessment, her efforts have been marked by controversy. 



stand it?" or "mat was the hardest thing about reading for 
you this year?" At each step of the Rhode Island projea. the 
evaluation is being fine-tuned, as it was at Mark Twain. Ques- 
tions are dropped, added, or modified. 

The way that assessment can enhance the learning of sni- 
dents is illustrated in Rhode Island by the evaluation of speak- 
ing ability. An evaluation form serves the twin purposes of 
instruction and assessment. Proceeding on the assumption that 
the best way to get snidents to reach the expected outcomes 
is to familiarize them with the expectations, snidents see the 
criteria by which they will be rated. 

In fact, the children are asked to apply these criteria to one 
another, using the same evaluation forms by which they will 
be rated. In most cases, the speech to be rated is an oral presen- 
tation of a book report. The snident is supposed to be suffi- 
ciently conversant with the book to be able to deliver the re- 
port by referring to notes, not by reading a written text. One 
teacher found that snidents were somewhat reluctant to fill out 
the evaluation forms because they did not want to rate one 
another, especially if it meant saying something negative about 
a classmate. She encouraged them to think of the evaluation 
as "helping one another." The assessment consists of the fol- 
lowing questions, each of which is answered by circling a yes 
or a no: 

Did the swdent 

• speak so that everyone could hear? 

• finish sentences? 

• seem comfortable in front of the group? 

• give a good introduction? 

• seem well-informed about the topic? 

• explain ideas clearly? 

• stay on the topic? 

• give a good conclusion? 

• use effective cosnimes. picmres. or other materials to make 
the presentation interesting? 

• give good answers to questions from the audience?* 

After giving a report, a snident has only to sort through the 
evaluation sheets marked by classmates to discover what he 
or she must do to improve. The crucial point here is that the 
strands of instruction and assessment are so interwoven. The 
Rhode Islanders are trying to help create classroom assign- 
ments that both support the growth of students in the five speci- 
fied literacy areas and prepare them for assessment, as the 
evaluation form in speaking does. The most promising prac- 
tices will be documented so that they may be shared among 
the teachers. 



THE DEFFICULTY of banishing the glitches ft-om al- 
ternative assessment may be sensed ft-om a glimpse 
at England's attempt to develop sets of "standard 
assessment tasks" that are to be used nationally for 
the first time in conjunction with the new national curricu- 
lum. Despite England's considerable experience with alteraa- 
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tive assessment, that nation's efforts have been marked by con- 
troversy. One teacher told of his fiustration in carrying out 
the pilot assessment in science for 7-year-olds last spring. He 
said: 

Close observation of the pupils was necessary thrx)ughout. 
It was difficult deciding whether Dionne was counting with 
her fingers under the table or mentally sotting through the num- 
ber bonds that she genuinely knew. During discussion about 
floating and sinking, it required considerable attention to as- 
sess the pupils against the 13 states of attainment.^ 

It took more than 35 minutes to test just four students on 
the scientific floating exercises alone, according to this teacher, 
and still the results were inconclusive. The proposed number 
of tasks to be performed in England's national assessment has 
been reduced by as much as two-thirds because the assess- 
ment, as originally conceived, demanded too much time. Sm- 
dents waiting to be assessed and those already assessed were 
supposed to work on their own while the teacher was off in 
a comer of the room assessing their classmates in smaU groups. 
But after snidents ended up being left unsupeirised for long 
periods - and sometimes disrupting classmates who wer« be- 
ing assessed - some schools sought to scrape up money to 
hire substinites to oversee the classes while regular teachers 
were conducting assessments. There is even talk now in Eng- 
land of insening some multiple-choice questions into the as- 
sessments to speed up the process. 

The matter of time is also on the minds of those developing 
the alternatives in Rhode Island. One day, as they sat around 
evaluating yet another pilot assessment, the assessors won- 
dered whether they should continue to review portfolios only 
in the presence of the snidents. "It might have been good to 
have had five or 10 minutes with the portfolios before the sm- 
dent walked into the room to discuss it," said one assessor, 
raising the prospect of adding even more time to a job that 
some believe lasts too long already. 

Subsequently it was decided that each portfolio would be 
reviewed briefly before the smdent arrived. This step would 
no: only increase the amount of time required for the assess- 
ment but would also mean that the portfolios would have to 
be more self-explanatory and more selectively assembled. In 
preparation for this change in policy, teachers were to recon- 
sider the kinds of pieces that should go into the portfolio so 
as to reduce the amount of peripheral material. After all, a 
portfolio should not be like a kitchen gadget drawer, so chock- 
a-block with unrelated items that locating the corkscrew be- 
comes a frustrating quest. 

Among the sorts of writing to be weighed for inclusion in 
the portfolio were: 1) a favorite piece of writing of any genre, 
2) a set of revisions showing the evolution of the writing proc- 
ess, 3) a creative and/or informative piece that responds to 
literanire that the snident has read, 4) a piece illustrating the 
student's undersunding in a particular content area, and 5) a 
wild card that the snident wants to include for whatever reason. 

56 



DECEMBER 1991 



279 



Since the content of the portfolio depends on what has been 
generated by assignments, the onus is on teachers to teach in 
ways that lead students to produce samples of appropriate 
work. At the same time, teachers must guide students in mak- 
ing proper selections from the various pieces they have writ- 
ten in each category, and they must prepare students to re- 
spond to an assessor in ways that will help him or her make 
a proper assessment. Ideally, the steps leading to an assess- 
ment and the assessment itself will be a learning experience 
for smdents. 

This responsibility can be forbidding to a. teacher unac- 
customed to teaching in this manner. Judy Wood, a third-grade 
teacher in Rhode Island, who is in her 28th and probably fi- 
nal year of teaching, can now reflea on her participation in 
the development of the alternative assessment with a litde less 
anxiety than she and her colleagues felt at the outset of the 
project. She said: 

At first, when they were asked to get involved, the teachers 
thought they would be given something that had already been 
completed. They were uneasy when they realized that they were 
getting involv^ in something that was only in the develop- 
ment stage. After a while, they understood that there was not 
a finished product for them and that their input would count 
and that there was a big, messy ingredient - the papers that 
were needed from the children for the portfolios. Finally, this 
fall, we were organized enough to know what was needed and 
to help the teachers in the first week or two of school so that 
those who were interested in joining in could surt getting the 
folders going. 



THE WAYS in which both learning and assessment 
might not be fully served were illustrated in one of 
the sessions in Rhode Island by a smdent who was 
asked to show her best work. (There has been some 
agonizing, incidentally, over possible differences in responses 
when children are asked for their "best" work as opposed to 
their "favorite" work.) She kept displaying examples from 
the beginning of the school year, 
although as a third-grader her 
work had presumably improved 
during the term. Difficulties arose 
in assessing other students when 
their math portfolios contained 
only the final step in solving a 
problem and showed litde evi- 
dence of how that point had been 
reached. 

In yet another instance, the as- 
sessor's task was made more dif- 
ficult by what was not contained 
in a smdent's math portfolio. "We 
wrote some word problems with 
circulars from Stop 'N' Shop," 
said the smdent, Stacy, reaching 
into a folder and pulling out a list 
of questions she had written us- 
ing the grocery prices in an ac- 
companying circtilar published by 



ERIC 



the store. One question asked, "If you had $8.99 and bought 
Head & Shoulders, how much would you have left?" The read- 
er had to find die price listed for the product in the circular 
and then subtract it from the amount in Sucy's problem. 

Stacy was asked why all her word problems involved only 
subtraction. She said that there were some that called for ad- 
dition, but she hadnt been able to find them. When asked about 
her use of math in other subjects, Stacy puUed from her port- 
folio pages of graphs that she had drawn comparing tempera- 
mres on various dates in Rhode Island and Hawaii, apparent- 
ly reflecting work in social smdies. Stacy had nothing to show 
when she was asked about what she had learned of division 
and multiplication. 

During their reviews of the pilot assessments, the assessors 
questioned whether even the settings might have affected the 
assessments, because some sessions were held in large rooms 
where other assessments were also taking place, while others 
were held in smaller rooms in which only one assessment at 
a dme took place. 

Such concerns have arisen elsewhere. Researchers at Eng- 
land's Bristol University who studied assessments of per- 
formance tasks conducted last spring in 57 of the county's 
primary schools wondered about the relationship of the set- 
tings to the comparability of outcomes. In some instances, 
smdents being assessed were regularly interrupted by com- 
ments directed at them by classmates working elsewhere in 
the room. At some other sites, the smdents being assessed 
were the only ones in the room and had the full anention of 
the assessor.' 

Despite the existence of models on which to build, diere 
is clearly much about alternative assessment that remains 
problematic. There are many hurdles that teachers must learn 
to leap before they assume responsibility for external assess- 
ment, as indeed they must if the activity is to be financially 
viable. 

Nor can it be taken for granted diat teachers will easily shift 
to alternative assessment. Many teachers do not fully under- 
stand the intricacies of die kinds of examinations widi which 

they have been working until 
now. Most states require no train- 
ing in assessment as a part of 
teacher certification - and, "even 
when assessment training is of- 
fered to teachers, it typically 
fails to provide the kinds of 
knowledge and skills needed to 
produce assessment literates.*^ 
There is promise in what has 
been happening in such endeavors 
as Arts PROPEL in Pittsburgh, a 
program in which teachers of mu- 
sic, of the visual arts, and of writ- 
ing have been altering their teach- 
ing to weave a blend of instruc- 
tion and assessment that enables 
both smdents and teachers to be 
reflective throughout the process 
of creation. But money from the 
Rockefeller Foundation and ex- 
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pertise from Harvard University and from ETS have periiaps 
made this venture easier to carry out than it might be in schools 
without such financial and human resources on which to draw. 

Another matter is seldom mentioned. Those who have cit- 
ed the need for equity in their rush toward altemadve assess- 
ment should recognize that students who score poorly on the 
much-maligned norm-referenced tests with their multiple- 
choice responses are not necessarily going to perform better 
on the alternatives. In fict, there is reason to suspect that the 
weakest students could look even worse — though they may 
avoid the embarrassment of being ranked and compared on 
a numerical scale. 

In England in the late 1980s, when the assessments that make 
up the General Certificate of Secondary Education were 
changed to put more emphasis on performance tasks (which 
are assessed by classroom teachers) and less on written an- 
swers, the gaps between the average scores of various ethnic 
groups increased rather dian narrowed. «> While such findings 
should not dampen the ardor of those eager to court alterna- 
tive assessment, the suitors must nonetheless be realistic and 
recognize that the bride has imperfections. 

Then there are issues related to the significance of being 
able to perform a given task. As I pointed out above, it seems 
reasonable to let students know that they will be examined on 
their ability to play a certain composition or execute a specif- 
ic athletic feat. But what about when it is less clear that the 
performance of the task is indicative of understanding that is 
transferable and not merely the result of memorization that 
barely outlasts the assessment? 

For example. New York State has used the same tasks for 
three years to assess the manipulative skills of fourth-graders 
in science. Each year a new group of fourth-graders goes to 
the same five testing stations as did fourth-graders the year 
before, where their skills are assessed in: 1) recognizing the 
physical properties of an object, 2) predicting, 3) inferring, 
4) creating a classification system, and 5) writing a generaliza- 
tion. The main reason why the 
tasks and the equipment have 
stayed the same for three years - 
changes are scheduled for 1992 
- is that making changes and 
training teachers to conduct new 
assessments are expensive. 

Fourth-grade teachers through- 
out New York know which tasks 
are to be assessed, and there is 
nothing to prevent them from 
coaching their students for the 
performance that occurs each 
May. Douglas Reynolds, who 
oversees this science assessment 
for the state department of edu- 
cation, said that the tasks - such 
as knowing how and when to use 
a thermometer or a ruler or an 
equal-arm balance - are widely 
transferable, and, even if children 
practice them in advance, all they 
are doing is learning what they 
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ought to know. He was quick to add, though, If we weren'^^ 
working on a shoestring and if I bad an infinite pool of items^jjjP 
and if I could go back and train people every year, then the 
tasks would not be the same each year." 

Worries of this sort are not unique to science tests for fourth- 
graders. At medical schods using altemadve assessments there 
has been concern about the impact of discussions between stu- 
dents who have taken the examinadons and those who are wait- 
ing to do so. This worry arose in connection with a lengthy 
assessment that could accommodate only small numbers of stu- 
dents at a time, as they took turns diagnosing ooock patients 
who simulated symptoms. 

Expense and time may turn out to be the brakes on the al- 
ternative assessment movement, both for the development of 
instruments and for their use. But thumbing through a portfo- 
lio with a student or watching a student perform a task - what- 
ever the psychometric worth of such assessments - adds a 
degree of intimacy that can be refreshing in an age of de- 
personalized appraisal. 

For instance, when the assessment in Rhode Island shifted 
from little William's reading to his writing, he was asked to 
show and discuss-the story he considered his best. Asked what 
he found most troublesome about writing, William — the proud 
author of a story about a nobleman who has a feast for his 
friends - did not pause for a moment, issuing a response that 
is universal in its simplicity for authors of all ages: Hlie-hardest 
part is covering the blank pages." And so it goes with alterna- 
tive assessment. 
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Reprinted with permission from Elsbeth Bellemere, Curriculum Director, 
Scarborough, Maine Public Schools. 



Assessment Recordkeeping 
in a Non-Graded 
ELSBETH Developmentally-Based 

BELLEMERE t>^^^^«* 

JEANNE KING Frogram 

The Scarborough SduxA System has adopted apolicy calling 
for 'Instructional Choices'. The policy reads, in part: 'Recog- 
niang the diversity of its students and their differing learning 
styles, the Board shall strive to provide a variety of programs 
designed to enhance learning. ' 

To implement this policy, Scarborough has created a non- 
graded, developmentally-based primary level program known 
as the GOLD (Grouping for Optimal Learning Development) 
Program. As its framework and as part of its assessment 
process, the program relies heavily on Jean Piaget's principles 
of cognitive development and the structure of thinking. 



In my classroom there 
are children ranging 
in age from five to 
eight years old. We 
have no text books. 
We have developed 
sets of criteria which 
a student must master 
before he or she can 
move on to the next 
level. 



Jeanne King, who teaches at the Eight Corners 
School, described the various components of her 
assessment process: 

"What's important about the Piagetian framework is that it 
promotes matching the child to the curriculum. As a result, the 
assessment becomes tied to that curriculum and what kids really 
do. In the program, we have a three-year rotation of themes: 
farm, forest, ocean community. We fit the science and social 
studies and other concepts in where they are appropriate. 

In my classroom there are children ranging in age from five 
to eight years old. We have no text books. We have developed 
sets of criteria which a student must master before he or she can 
move on to the next level. 

The cornerstone of the assessment is constant and organized 
recordkeeping. The teacher is not the center of attention. I'm 
the facilitator/observer so I'm constantly observing, taking 
notes, going through students' work. We save their work to 
check progress over time and to keep as a record of the child's 
accomplishments. 

For example, one way I get to know students is by reading 
with them individually once or twice a week. When I do this, 
I note the strategies the child is using and write comments on the 
appropriate check list. The GOLD teachers created the report 
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card and the math and literacy recordkeq>ing (see pages 31 and 32). 

As part of this record-kiseping students keep their work in a variety of folders-such as 
math, handwriting, creative writing, science, center work. Students keep'lisu of what 
they've read during the year so I have a chance to see what they're doing and talk with them 

about it. ' 

A key piece in our assessment is the Piagetian analysis of each child's cognitive skills (see 
seriation example and summary on pages 33 and 34). In the fall and spring, the town hires 
substitutes so that the teachers can test the children one^n-one. For me, this is important 
because the teacher has to be constantly aware of these thinking processes when working with 
kids. We use the Individual Piagetian Summary to record what we observe about a student's 
abilities in these areas at the beginning of the year. During the year, when a child reaches 
a new point, I note it and record it on the Piagetian Summary. 

The record-keeping is the most time-consuming part but I believe in it because I know the 
children better and, in this program, it becomes an on-going assessment over three years. One 
of the biggest changes I've observed is a decrease in the level of student frustration and an 
improvement in self-esteem and student behavior. I think this happens because our process 
supports what kids can do, not what they cannot do. This program puts more responsibility 
on the teacher for curriculum and assessment. I have to ask, 'How do I know these Mds know 
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NOTE: Thcst txamples art gxetrpts 
from tha actual forms 



YEAR TEACHER 



STUDErfT'S NAME 

LITERACY SKILLS . GOLD PRIMARY AMD INTERMED IATE RECORD 
iQlfir Primary Transitional 



□ □ Rtads indtp«nd«ntiy utins a widt rangt of books 

□ □ Incrtasas sight vocabulary 

□ □ Continues to apply word strattgits with unknown words, usts conttxt to prtdict 

unknown words 

□ □ Stif-corrtcts whilt rtading 

□ □ Reads with comprthension characttrs, stqutnct, thtmt, motivation 

□ □ Prtdicts, makts Inftrtncts rtlated to story 

□ □ Identifies favorite types of books, authors 

□ □ Gathers information from non-fiction sources 







Writino 




□ 


Name 




□ 


Words 




□ 


Phrases 


□ 


□ 


Sentences 


□ 


□ 


Stories 


□ 


□ 


Formulates ideas 


□ 


□ 


Composes complete thoughts 


□ 


□ 


Edits and refines material 
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BEST COPY AVAILABLE 



Ymt 



SniMnf a nmm 



Taaciw 



NOTE: IhcM wsmptts an •MwprW 
from th« actual loon* 



60L0 PflBMAIIY MATH STUDENT XECORO 



FRACTX)NS 



Cutt wtwit/erauDt into para: labda 



1/2 

a 



1/4 

a 



1/3 
□ 





RacoQfiUas synvrMtry 


□ 


a 


□ 




Raada symbola 


□ 


a 


a 




Writ** 


a 


a 


a 


2. 


OPERATIONS 




Conciata 


Symbols 






orpicttTCS 






Caocsfli 


with ivmboli 






SUBTIUCnON 










Counts back mcntaUy 


□ 


□ 


□ 




Stparatts Mts 


□ 


□ 


□ 




diftwancts from 10 


□ 


□ 


a 




difftrtncas from 20 


□ 


□ 


□ 




Computes diffartncts from 99 


□ 


□ 


□ 




without regrouping 


□ 


□ 


□ 






□ 


□ 


□ 






□ 


□ 


□ 




Uses *0* in subtraction 


□ 


□ 


□ 




Knows minus is rtvt/s* of phis 


□ 


□ 


□ 


3. 


MONCT 










Recognizes coins 


a 


□ 


□ 




Krwws values of coins 


□ 


a 


□ 




Counts coirxs 


□ 


a 


□ 




Exchanges coirts 


□ 


□ 


□ 




Makes cfiar>ge 


□ 


a 


□ 


4. 


GEOMETRY 




Concrete 








or picttires 


Symbols 








with ivmboli 


(♦ . -1 




Sorts, names plane shapes (201. 


a 


a 


□ 




solids (301 


a 


□ 


□ 




Recognizas shapes/solids in tfia environment 


a 


a 


□ 




Makes shapcs/soWs 


a 


□ 


□ 




Reproduces panems using shapes 


a 


□ 


□ 




Makes and descnbes repeated pancms/tesscllations 


a 


a 


□ 
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What Research Says 




Evaluatin 
Elementary 
Science 
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Since the advent of the activity- 
oriented science programs of the 
1960s, educators have been concerned 
about the need for appropriate assess- 
ment instruments. Today, New York 
State has addressed that need with an 
innovative system that includes au- 
thentic science assessment. 

Make a Change 

The changes in New York didn't hap- 
pen overnight. For decades (since 
1958), elementary science instruction 
had followed guidelines specifying 
topics for each grade level. There was 
no state assistance with local program 
development, nor were there any state 
elementary science tests. 

Then, in 1985, four important 
changes took place. First, a new state 
syllabus established outcomes for sci- 
ence content, skills, and attitude. Sec- 
ond, teacher supplements to that syl- 
labus featured sample program 
materials. Third, an assessment de- 
vice, the Elementary Science Program 
Evaluation Test (ESPET) was pro- 
posed to assess the effectiveness of 
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the science instructional program in 
each school. Fourth, the state devel- 
oped a statewide elementary science 
mentor network of experienced teach- 
ers and administrators trained to sup- 
port the development of local school 
programs and to administer and score 
ESPET. Without all four components 
of the system (syllabus, program, as- 
sessment, and mentor network), it is 
unlikely that positive, sustained 
change in elementary science could 
have occurred. 

In May 1989, 211,000 fourth-grade 
students, handicapped and non- 
handicapped, enrolled in public and 
nonpublic schools in New York, took 
the first ESPET. Based on the out- 
comes of Levels I and II of the new 
syllabus (grades K-4), the untimed, 
45-item multiple-choice objective 
test measured science content and 
inquiry skills. In addition, many 
school districts administered one or 
more of the optional survey compo- 
nents that included a 20-item sci- 
ence attitudes survey, a 25-item pro- 
gram environment survey, a 36-item 
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teacher survey, a 32-item adminis- 
trator survey, and a 12-item parent/ 
guardian survey. 

Testing Manipulative Skills 

New York State was the first in the 
nation to administer a large scale ma- 
nipulative skills test statewide to all 
students' at a given grade level. The 
test consisted of tasks at a series of 
five locations (stations) set up in a 
room by the school's elementary sci- 
ence mentor (ESM). 

Station setups, equipment lists, 
and verbal directions were standard- 
ized across the state according to an 
"administrator's guidebook" that each 
mentor had received in a workshop. 
These guidelines enabled the test ad- 
ministrators to check materials 
quickly, just before students moved to 
the next station, and helped make the 
test valid across the state. Figure 1 
shows the materials list, station setup 
diagram, and directions for preparing 
"Station Two— Water on Objects." 

Figure 2 shows a classroom setup 
for 25 students. Students were able to 
move from station to station with 
minimal instructions. Because they 
were kept busy at their own stations, 
the children had little temptation to 
observe others' work. Also, the sta- 
tions were staggered, so any "observ- 
ing" would be of some other task. 

TVial Testing 

We began by collecting existing as- 
sessment ideas from as many sources 
as possible. To develop the skills test 
of ESPET, we examined the following 
sources: 

• the Second International Science 
Study (SISS), 

• the Assessment of Performance. 
Unit(APU), 

• the National Assessment of Educa- 
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tional Progress (NAEP), 
• and commercial material and cur- 
riculum projects (SCIS and SAPA, 
for example). 

Doran's (1990) review of "practi- 
cal" tests horn various national and 
international surveys and Hein's 
(i;j90) collection of papers on the as- 
sessment of hands-on programs also 
came into play. 

The SISS had developed and ad- 
ministered skills tests for fifth grad- 
ers, so we modeled our initial tasks 
on their material and modiHed tasks 
from other sources accordingly. Our 
model asked a series of questions con- 
nected to a set of materials or a com- 
mon problem; involved a set of safe, 



easily accessible material^' and equip- 
ment; and included a set of scoring 
guidelines with point valuer allocated 
to each question. 

Producing a set of tasks for state- 
wide testing purposes that will work 
well in a wide variety of classrooms 
takes at least two years to develop and 
reHne. Because these tests are assess- 
ing the outcome of a specific curricu- 
lum (school, district, or state), the 
goals and objectives for that curricu- 
lum must exist in a published form 
and be considered at each stage of 
development 

A team consisting of the regional 
elementary science mentors from 
western New York, specialists from 



the science and testing bureaus of the 
state's Education Department, and a 
university science educator was 
charged with shepherding these tasks 
from inception to statewide imple- 
mentation. The creativity and wisdom 
from their teaching and mentoring 
experiences wcr?! essential at every 
stage of development 

As each new task was proposed, 
one mentor was assigned to refine it 
and modify it to a common format If 
it seemed workable, the mentor tried 
the task with a few students, check- 
ing the materials' suitability and the 
time needed to complete the task. Af- 
ter making necessary revisions, the 
team tried a sample of tasks with 10- 
15 students. 

At that point, the pool of tasks were 
analyzed according to skills assess^jjj^ 
grade level of language used, matenfl^ 
accessibility and "mess quotient" Af- 
ter screening and subsequent revi- 
sions, these tasks were combined in a 
mock-up with the same number and 
length of time as desired for the state- 
wide test The mock-up was tried in 
six to eight schools (one class each) 
with verbal directions, test booklets, 
timing, and so on. The mentor (or a 
classroom teacher) administered the 
test Other experienced mentors and 
representatives of the state Education 
Department observed the session, 
looking for difHculties with reading, 
equipment and time constraints. 

We tested about twice as many 
tasks as we would need for the ESPET. 
Based on student responses and on 
observations by teachers and test ad- 
ministrators, we assembled recom- 
mendations for modification. The 
items that assessed a broad range 
skills and had the best testing charal^F 
teristics were then assembled for 
statewide field testing. 



Figure 1. Station Two's materials list, setup diagram, and directions. 
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1 1 . Make sure ttiat fresh sets ol papers A, B, and C are available for every student 
't be tested at the station..'^ 



will 



rnir> ^ SCIENCE AND CHILDREN 

ummmmmm 



81 



NOVEMBER/DECEMBER 1992 




ERIC 



sample ot 1,05^ tourtn graders) to 
represent the diversity of schools 
within the state. These schools were 
in . ited to participate in the next 
step — ^the statewide field test. This 
field testing was done exactly as it 
would be in subsequent statewide 
implementation, in terms of direc- 
tions to be read by the test adminis- 
trator, questions asked, student in- 
structions, arid materials and 
equipment kits. Following the field 
testing, we made final adjustments to 
the instrument and also established 
norms for interpreting the results. 

Scoring Procedures 

Because teams of teachers at each 
building or district would be scoring 
the tests, we developed a clear and 
easy-to-use set of scoring procedures: 

• a smcill number of points were al- 
located to each item; 

• a scoring guide clearly stated the 
criteria for assigning points; 

• and examples of answers were 
given for each point value. 

A set of scoring procedures was 
drafted, revised, and trial-tested prior 
to use in the ; a.ewide field test. Each 
station had several questions for the 
student to answer. Each question had 
a maximum point value between one 
and three. As the maximum number 
of points possible for any question 
was three, the deliberation by the rat- 
ers was minimal. The point value for 
each station was four or five, with a 
maximum score on the skills test of 
22 points. The rating guide provided 
as much help as possible in listing 
sample correct answers, but scorers 
were instructed to give appropriate 
credit for answers conveying the same 
general meaning as those listed. Such 
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Figure 2. Sample classroom setup. 

a provision is critical if scorers are to 
avoid the "single answer mentality" 
that pervades assessment 

The rating/scoring guide is orga- 
nized into three parts — an overall set 
of procedures, criteria and acceptable 
answers for each item, and sample 
student answers illustrating each 
point value for each item. 

The criteria and acceptable answers 
for the first question at Station Two 
are shown below, along with a stu- 
dent example from the rating guide. 
What happened to the drop of watet on 
each piece of paper? 
Criterion: The student correctly describes 
what happened to the drop of water on 
each piece of paper. 
Maximum score: three points 
Samples of acceptable answers: 
On papers A and C, the drop of water 

was absorbed 

soaked in 

spread out 

got bigger 

expands 

fills up/makes squares or blocks 

goes through 
On paper B, the drop of water 

was not absorbed 

sits on top 

stays in ball 

stays the same 

doesn't spread 

stays a drop 

won't go through 

bubbles on top 
Number of Credits 

What happened to the drop of water on 
each piece of paper? 

+ 1 On paper A, the water drop soaked in. 
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soaked in. 

Comment The student correctly de- 
scribed what happened to the drop of 
water on papers A and C, but not on 
paper B. 

Did Scorers Agree? 
Seven teachers who had been train- 
ers for tiie scoring workshops trav- 
eled to Albany to re-rate 3,949 of the 
May 1989 ESPET manipulative skills 
tests. The number of discrepancies 
was tallied by school, by station, and 
by item within each station. 

Overall, tiie two ratings agreed 
more than 90 percent of the time 
The level of agreement, however, did 
vary considerably by **.sk and by ques- 
tion within task. For example, an item 
on measuring mass involved only 
clear, simple scoring criteria and led 
to few discrepancies; while a request 
that students "write a statement about 
electricity" called upon scorers' sub- 
jective judgement and resulted in a 
high number of discrepancies. 

A compilation of the scoring diffi- 
culties and recommendations for 
remediation was sent to schools 
whose papers wer .. re-rated. Schools 
with more than 10 percent total dis- 
crepancies were recommended for a 
scoring workshop. 

The Results Are In 

Schools were required to send to the 
state Education Department a sum- 
mary of their ESPET performance on 
a form called tiie "Comprehensive As- 
sessment Report" (CAR). Similar 
school summary data were collected 
on CAR for other state-administered 
tests. These data were then compiled 
so schools could see how their perfor- 
(continued on page 63) 
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(Evaluating, 

continued from page 35) 

mancc compared with other schools 
in their county, with similar-sized 
schools, and with all the state's schools. 

Performance on the manipulative 
skills test at each school was also 
viewed within the context of the state- 
wide data. The data shown below re- 
flect the percentage of students who 
answered correctly for each question 
of the skills test (May 1989 data). 



on the measuring cup. This series of 
steps made volume measurement 
more complex than the other mea- 
surements. Some students incorrectly 
used the centimeter scale or the ther- 
mometer for measuring volume. 

At the second statioa students had 
to record their observations of water 
drops placed on three kinds of paper. 
They were very successiul at this task; 
86 percent earned full credit The sec- 
ond item asked students to predict 
what would happen to a water drop 






Looking at the data by station and 
by item, we can make some generali- 
zations. At the first station, students 
scored relatively well on simple mea- 
suring tasks, with 70-80 percent cor- 
rectly determining mass, length, and 
temperature with a centimeter scale, 
thermometer, and pennies (..onstand- 
ard mass units). 

Measuring volume proved a bit 
more difficult; students had to pour 
water into an unmarked glass or mea- 
suring cup and then compare with 
either a line on the glass or the scale 

O 
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on a piece of paper within a sealed 
clear-plastic bag. The "unknown" pa- 
per was similar to one of the "tested" 
papers, and students were relatively 
successful with this prediction (77 
percent correct). The third item 
probed the reasoning behind their 
prediction; fewer children were able 
to do this than predicted correctly 
(63 percent). 

At the third station, students were 
asked to sort objects (seeds) into two 
groups based on a common property 
and then state the shared characteris- 
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tic for each group. Approximately 80 
percent of the students sorted the 
seeds into two groups and described 
the shared characteristic 

At station four, most students 
benefitted from previous work witi) 
batteries, bulbs, and wires. They were 
skilled in checking the five objects 
(wire, paper clip, spoon, foil, and 
toothpick) in a circuit to see which 
one made the bulb light The scoring 
for this item was as follows: 

• 3 points — ^5 objects tested correctly 

• 2 points— 4 objects tested correctly 

• 1 point — 3 objects tested correctly 

• no score — less than 3 objects tested 
correctly 

Most students were successful in test- 
ing these objects. The second item — 
writing a statement about electricity 
and all the objects tested— was con- 
siderably more difficult The average 
score for this item was 59 percent 
Many students wrote only about the 
objects that made the bulb light rather 
than including a statement about all 
the objects. 

The fifth station required students 
to infer properties of objects within a 
sealed box from the senses of hearing 
and touch. The students were more 
successful inferring the shapes of the 
objects (92 percent) than their mo- 
tion (72 percent). "Shape" is a widely 
used property, more familiar than 
motion. Another item at the station 
asked, "What is another property of 
an object in the box?" This also proved 
to be difficult (49 percent correct). 
Finally, students were asked to ex- 
plain how they estimated the number 
of objects in the box. As most stu- 
dents "heard" collisions between ob- 
jects, they » 'ere very successful at this 
task (86 percent). 

Follow a Leader 

It is possible to assess manipulative 
skills associated with an elementary 
science program, but it does not hap- 
pen in a vacuum. In New York State, 
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we built upon a revised syllabus, man- 
dates for local program development 
and state level assessment, and an 
elaborate network of teachers train- 
ing teachers (mentors). We hope in- 
dividual teachers, school districts, and 
states will continue building upon this 
foundation. 
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Science for All: Getting It Right 
For the 21st Century 



Kenneth M. Hoffman and Elizabeth K. Stage 



National standards in curriculum, teaching, and 
assessment— to be published in the fall of 1994— 
will translate the vision of "science for all" 
into concrete direction for achieving it. 



In December 1892. 18 men met at 
the University of Chicago to advise 
the "Committee of Ten" on science 
preparation needed for college 
admission. The consultants were 
teachers from high schools and prep 
schools and faculty at public and 
private colleges and universities. Their 
consensus was that at least one year of 
biology, followed by one year of 
chemistry and one year of quantitative 
physics, would best prepare young 
people to grow up to be just like them 
(National Education Association 1894). 
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Recently, in December 1992, 600 
men and women — mathematics and 
science teachers, supervisors, state 
coalition and systemic initiative direc- 
tors, assessment reformers, governors' 
education aides, and others — gathered 
in Washington, D.C., to discuss 
preliminary working documents cf che 
National Committee on Science 
Education Standards and Assessment 
(NCSESA). These materials address 
"Science for All," a challenge that our 
nation is finally, 100 years later, ready 
to embrace. 
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The Legacy of the Committee of Ten 

There's a lot to like about the 1892 
reports to the 10 college and university 
presidents, including a recommenda- 
tion "that the laboratory record should 
form pan of the test for admission to 
college." What's not easy to like is 
that the content recommendations set 
the high school curriculum that 
remains in place today for nearly all 
students. This has led to the current 
situation: some science for some 
students. 

As the body of scientific knowledge 
has exploded, high school courses 
have become cluttered with so much 
new vocabulary— often exceeding that 
of foreign language courses — that 
terms can only be memorized rather 
than understood. To prepare students 
for this onslaught of disconnected 
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facts, junior high courses have often 
imitated high school courses, with 
levels of abstraction and quantification 
that go beyond the intellectual 
capacity of young people. Junior high 
students, too. have leaned to succeed 
by memorization. Since memorized 
work is easily forgotten, teachers at 
each level teach as if students* minds 
are empty. Thus, the expectations for 
elementary school children are usually 
minimal: "Keep their curiosity alive." 

The elementary curriculum depends 
largely on the interests of teachers, 
only a quarter of whom feel "well 
qualified" to teach science (Weiss 
1989). Is it any surprise, then, that 
although 70 percent of elementary 
students say they are interested in 
science (Weiss 1989), by the time they 
reach high school, science enrollments 
drop by more than one half each year? 
Only 20 percent of high school 
smdents nationally take the fmal 
course in physics recommended in 
1892 (Blank and Dalkilic 1990). 

Challenges to the Status Quo 

What's wrong with that? Since only 
3 or 4 percent of the work force is 
engaged in science and engineering 
(U.S. Department of Labor 1992). 
why do all of our citizens need to learn 
science? Concerns about competitive- 
ness in the global economy are fueling 
the renewal of science and mathe- 
matics education. The business 
community demands entry-level 
workers who are able to think and 
solve problems. Regardless of our 
relative international rank, informed 
citizenship in the year 2000 requires 
that all people have a substantially 
greater understanding of science. 
Recall this fall's ballot initiatives in 
several states, or consider the super- 
market dilemma — "Paper or plastic?" 
Increasingly we are confronted with 
questions for which scientific informa- 
tion and ways of thinking are neces- 
sary for informed decision making. 

Finally, a well-kept secret: science 
is one avenue through which humans 
can seek understanding of our place in 
the universe. The personal fulfillment 



We need national 
standards to 
highlight and 
promote the 
best practices 
of the heroes 
who do what 
needs to be done 
despite the norms. 



and excitement that science has to 
offer benefit everyone. For these 
reasons, scientists and science educa- 
tors are taking advantage of the 
current attention on national education 
goals to do a better job this time 
around. 

Science for All Americans is not 
only the name of an influential book 
issued by Project 2061 — the far- 
reaching effort of the American Asso- 
ciation for the Advancement of 
Science (AAAS) — but also its goal 
(Rutherford and Ahlgren 1989). 
Project 2061 was initiated in 1985, a 
year in which Halley's Comet came 
close to the earth, and named for the 
year in which the comet will return. 
The project delineates the science that 
people whose lives span those years 
will need to achieve scientific literacy. 
Taking the long view, it defines 
"science" broadly, to include the 
natural sciences, as well as the social 
sciences, mathematics, and tech- 
nology. Project 2061 is taking a 
decade to produce curriculum models 
and blueprints for teacher education, 
assessment, and other systems that 
need to change to realize the vision. 
This slow, deliberate pace respects 
the premise tliat you cannot create 
an airplane by adding wings to an 
automobile. 

In contrast, the National Science 
Teachers Association (NSTA) has 
crafted a more immediate solution. 
The 1892 Committee of Ten recom- 
mendation "that it is bener to study 



one subject as well as possible durin^j^ 
the whole year than to study two or 
more superficially during the same 
time" has led to the high school 
courses we have today. There is insuf- 
ficient time for the introduction of 
concepts, starting with an experiential 
base and leading later to absoaction 
and formalization. The learning 
process has been compressed into too 
short a time frame for meaningful 
understanding to occur. To improve 
this situation, NSTA's Scope, 
Sequence, and Coordination Project 
(SSQ recommends that biology, 
chemistry, physics, and earth science 
(a significant omission of the 
Committee of Ten) be taught each 
year, starting in the 6th or 7th grade 
and continuing through 12th grade 
(Aldridge 1992). The SSC slogan, 
"Every Student, Every Science, Every 
Year." is a short-term version of what 
Project 2061 envisions. 

Why Standards? 

With a far-reaching vision for the 
future and short-term solutions being 
implemented in schools today, why do 
we need national standards for science 
education? First, standards are criteria 
by which judgments can be made. 
They need to be based on a vision, to 
be sure. But they must also address 
characteris'^cs of curriculum design so 
that local educators can select what 
they want their students to learn: one 
of Project 206rs curriculum models; a 
particular version of Scope, Sequence, 
and Coordination: or another option. 

What is needed from the national 
level is guidance for making those 
decisions. To understand the need 
for direction from the national level, 
one need only watch a mathematics 
teacher arguing for a better system of 
assessment on the basis of Curriculum 
and Evaluation Standards for School 
Mathematics (National Council of 
Teachers of Mathematics 1989). 
Or ulk with a textbook publisher 
who wants to make wise selections 
among the plethora of material in 
current K-12 science cuiriculums. The 
banner put fotward by NCTM's Stan- 
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dards enables everyone to move in the 
same direction, assured that the risks 
they take to improve mathematics 
education will be supported by poli- 
cies and practices throughout the 
system. 

Of course, good things are 
happening in science classrooms today, 
even without national standards — but 
they happen because of the heroes who 
do what needs to be done despite the 
norms. Many generous elementary 
teachers, for example, spend their own 
money on science supplies because 
they know that their students learn best 
by investigating. Middle schools often 
use science programs with relevance to 
their students' lives instead of being 
merely a practice for high school. And 
some high school teachers, ignoring 
the vocabulary-dense syllabus, 
encourage student inquiry into ques- 
tions of their own. 

We need national standards to high- 
light and promote the best practices of 
these heroes. We must make their 
curriculums the exemplars — the core 
of teacher preparation programs, the 
models for instructional materials and 
assessments, and the basis on which 
science programs are judged. We need 
to recognize and encourage the school 
principals who find money in their 
budgets for field trips, the parents 
whose bake sale proceeds purchase 
science equipment, the authors who 
write materials that cannot possibly 
satisfy the divergent criteria of 22 
stales and thousands of local districts, 
and the publishers who are pioneering 
in authentic assessments despite the 
lucrative market for multiple«choice 
tests. These leadership efforts must 
become the goal toward which others 
strive. 

How Are Standards Being Developed? 

The National Research Council — the 
operating arm of the National 
Academy of Sciences and the National 
Academy of Engineering — agreed 
to take the lead in developing stan- 
dards for science education, at the 
request of the National Science 
Teachers Association and other profes- 



sional societies, the Secretary of 
Education, the Assistant Director for 
Education and Human Resources of 
the >}ation3l Science Foundation, and 
the Governors who co-chair the 
National Education Goals Panel. 

While scientists and science 
teachers are prominent in the process, 
teachers are a plurality on all commit- 
tees and working groups. Also 
involved are other educators, business 
representatives, and the public. To 
further rsctify the sins of omission of 
the 1892 committee, women and 
members of other groups currently 
underrepresented in the sciences 
(people of African-American, Latino, 
and Native American origin) are 
participating directly in the process. In 
a further attempt to ensure that the 
views of 18, or even 180 people, do 
not set the agenda for standards 
intended for all students, a plan for 
broad critique and consensus has been 
built into the process. 

Meeting for the first time in May 
1992, the National Committee on 
Science Education Standards and 
Assessment approved a plan to 
produce science education standards 
by fall 1994. The committee's charge 
to its working groups on curriculum, 
teaching, and assessment is: 

...to develop, in cooperation with the 
larger science, science education, and 
education communities, standards for 
school science. 

The standards, founded in exemplary 
practice and contemporary views of 
science, society, and schooling, will 
provide a vision of excellence to guide 
the science education system in produc- 
tive and socially responsible ways. 
Standards for curriculum, teaching, and 
assessment will be integrated in a 
single document. The standards will 
specify criteria to judge the quality of 
school science and to guide the future 
development of the science education 
enterprise. 

What Standards Are Being Developed? 

Science curriculum, science teaching, 
and science assessment standards are 
being created. Curriculum standards 
will define: 
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■ the nanire of school science 
experiences that exemplary practice 
and learning research propose are 
effective in producing valued science 
learning; 

■ the scientific information (facts, 
concepts, laws, theories), modes of 
reasoning, and proficiency in 
conducting scientific investigations 
that all snidents are expected to anain 
as a result of the experiences; 

■ the attitudes and inclinations to 
apply scientific principles and ways of 
thinking outside the formal educa- 
tional system that all students are 
expected to attain. 

Science assessment standards will 
define: 

■ the methods for assessing and 
analyzing student achievement and the 
opportunities that programs afford 
students to achieve the valued 
outcomes of science; 

■ the methods for achieving appro- 
priate correspondence between assess- 
ment data and the purposes that the 
data will serve; 

■ the characteristics of valid, reli- 
able science assessment data and 
appropriate methods for collecting 
them. 

Science teaching standards will 
define: 

■ the skills and knowledge teachers 
need to provide students with school 
experiences to achieve the valued 
science learning outcomes; 

■ the preparation and professional 
development teachers need to fulfill 
their roles; 

■ the necessary support systems 
and resources for effective science 
teaching. 

The National Science Education 
Standards will be descriptive, not 
prescriptive, in order to support 
thoughtful consideration and applica- 
tion. The curriculum standards will 
not prescribe particular courses, 
programs of study, or textbooks; 
assessment standards will not be an 
examination; and teaching standards 
will not be certification or licensure 
specifications. In each case, examples 
will illustrate the broad range of what 
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Prototype Activity: Matter (ist erode Levei) 



The tallowing is an example of how 
content standards can be taught and 
ev^uated. This passage is excerpted 
from: Natiortat Committee on Science 
Education Standards and Assessment. 
(December 1992), National Science 
Educate Standards: A Sampter 
(Washington, D.C.:t4atior\al Research 
Council), pp. 30-33. 

...Today, the teacher had planned to 
take the class for a walk arouxi the 
block ... [to] collect rocks for study.... 
The teacher told the students about the 
purpose of the walk and asked them 
what they thought they might find and 
where they might find them. Divided 
into pairs and equipped with a map of 
the block ... and a bag. the children 
circled the block, stopping to collect 
stones as they went. Back in the class- 
room, the students, in groups of tour, 
examined their rocks closely, using 
hand-held magnifying lenses. They 
were asked to think about how they 
might describe their rocks, making 
drawings if they wished, and then to 
sort their rocks into groups that made 
sense to them. 

Within iheir groups, the students 
discussed their obsen/aticns and 
agreed and disagreed about cate- 
gories. The teacher moved from group 
to group, listening to the discussions, 
asking for descriptions, pointing out 
interesting features, and querying the 
reasons for the groupings.... 

The next day. the teacher [asked] 
each group ... to explain the basis for 
the grouping of their rocks. Other 
students were asked to comment. The 
teacher picked up a new rock and 
asked that it be placed in the proper 
pile. After each group of four had 
completed its explanation, the teacher 
and iha class constructed a list of all 
the characteristics ... used in sorting 
the rocks. They discussed the relative 
usefulness of some versus others and 
talked about other tools that might be 
useful.... 

...This unit will continue. The 
students will pursue the study of rocks 
as well as other parts of the environ- 
ment. In the process, they will continue 
to study the properties and character- 
istics of objeas and materials and 
apply their abilities to observe, 
describe, and classify.... 



— Asa result of [many] activities [like 
these] students shoukl be able to 
damonstrats their understanding of 
fundamental ideas about objects and 
materials: namely, that 

■ ComnKsn objects have obsen«ble 
properties (size, shape, volume, and 
weight)1hat can be compared and 
measured ... [and] used to describe, 
group, and classify objects. 

In demonstrating their under- 
standing of these ideas, students 
should be able to classify or order 
a set of objects according to a speci- 
fied property, such as weight or 
volume.... and to devise one or more 
ways to classify or order a set of 
objects and ... to explain their classifi- 
cation scheme. 

■ Objects are made up of different 
kinds of materials. Materials have 
observable properties (color, texture, 
magnetic characteristics, and different 
behaviors when heated or cooled) 
that can be compared and measured. 
Such properties are useful in 
describing, grouping, and classifying 
materials. 

...Students should be able to group 
a set of objects according to the mate- 
rials from which the objects were made 
(wood, metal, glass, and clay).... [and] 
describe differences in the obsen/able 
properties of such materials. 

■ Materials can exist in different 
states (solid, liquid, gaseous). Each 
state has characteristic properties. 

...Students should be able to 
describe obsen/able properties that 
given materials have in common or that 
distinguish them from one another. 

■ Some properties of a material may 
change when it experiences extemal 
change; others do not. In particular, if 
ttie temperature of a sample of mate- 
rials is changed, the material may 
change from one state to another 
(liquid to solid, liquid to gas, and so on). 
However, the weight of an object 
remains unchanged when it is broken 
into smaller parts. 

...Students should be able to predict 
and describe the effects of tempera- 
ture changes on water or ice.... [and] 
to provide evidence that the weight of 
a sample of material remains the same 
even though its shape, locatio.i. or 
appearance may change. 



is possible, not define the one "best'^^B 

approach. 

While woridng groups in 
curriculum, teaching, and assessment 
convened separately in the summer of 
1992, their overlapping membership 
and common goal will lead to a 
unified document that will move the 
system of science education foi-ward 
in concert The first discussion docu- 
ment, prepared in October,' outlined 
guiding principles for the production 
of a complete draft by fall 1994. These 
principles delineate the territory of 
school science — somewhat broader 
than Scope, Sequence, and Coordina- 
tion, but narrower than Project 2061. 

Science for All 

The first principle. Science for All. 
takes an unwavering stand that "the 
science standards will define the level 
of understanding that all students — 
regardless of background, future aspi- 
rations, or interest in science — shou 
develop." The text foreshadows wh; 
will appear in subsequent drafts: 

... the commiBnent to "Science for All" 
implies inclusion not only of those who 
traditionally have received encourage- 
ment and oppominity to pursue science, 
but of women and girls, all racial and 
ethnic groups, the physically and 
educationally challenged, and those 
with limited English proficiency. 
Further, it implies attention to various 
styles of learning and differing sources 
of motivation. Every person must be 
brought into and given access to the 
ongoing conversation of science. 

Thus, the commitment to "Science for 
Air requires curriculum, teaching, and 
assessment standards that take into 
account student diversity vis-a-vis 
interests, motivation, experience, and 
ways of coming to understand science. 
The standards must define criteria for 
high-quality science experiences that 
include the engagement of all students 
in the full range of science content 
These experiences must teach the 
nature and process of science as welt^^k 
the subject matter and support die ^i^ 
notion that men and women of diverse 
backgrounds engage and participate in 
science and that all have a claim on this 
common human heritage. 
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The commitment to "Science for AH" 
has implications for program design 
and resource allocation at local, state, 
and national levels. 

The other guiding principles in the 
October discussion document delin- 
eate the territory of school science, 
distinguishing it from technology and 
engineering and from other ways of 
knowing. The position taken is that the 
national science education standards 
should be limited to the fundamental 
understandings and should offer selec- 
tion criteria that states, localities, 
teachers, and students can use to deter- 
mine additional subject matter to be 
studied. Such criteria will include: 
developmental appropriateness, expe- 
riential connections, contribution to 
students' ability to investigate and to 
make decisions, and being worth the 
instructional time and student effort to 
achieve understanding. 

These positions are elaborated in a 
December discussion document, 
which provides one or more prototype 
standards illustrating the interweaving 
of curriculum, teaching, and assess- 
ment. This document also character- 
izes the domain of science education 
standards, indicating the inclusion not 
only of the subject matter (ecology, 
energy, space, and so on) but also 
inquiry, decision making, and content 
(social, ethical, and historical). 

Critique and Consensus 

Parallel to the development of science 
education standards is a broad-based 
critique and consensus process. By 
working with the several science and 
science education communities 
(biology, chemistry, physics, earth and 
space sciences), we hope to overcome 
the fragmentation and territoriality that 
have characterized much of science 
education in the past. By involving 
those who have been left out — females 
(Association for Women in Science); 
members of racial and ethnic groups 
(American Indian Science and Engi- 
neering Society, Hispanic Secretariat, 
National Association of Black School 
Educators); and the physically chal- 



lenged (Foundation for Science and 
the Handicapped) — we will woik to 
enlarge the mainstream of the science 
and science education community. 

We distributed the December draft 
to a wider audience than the Oaober 
document The overwhelming senti- 
ment among the more than 5,000 
scientists, science educators, and 
educators with whom we have talked 
is concern that the public will not 
suppon the changes the standards will 
call for. The publishers and producers 
of insoructional materials and tests, 
with whom we have had two meetings 
each already, are also eager to see 
public support for hands-on, "minds- 
on" science programs. Similariy, the 
corporate community is willing to 
work for standards-based systemic 
change — and is poised to assist with 
expertise and funding. The National 
Governors' Association has put stan- 
dards-based, systemic reform at t;:s 
top of its 1992-93 agenda, with mathe- 
matics and science leading the way. 
Other policy groups havt given us 
similar endorsements. 

By extending the discussion to 
include the broader education, busi- 
ness, parent, and policy communities,* 
we hope to create a context of wide 
support for the science education 
goals. And by working with the public 
and private funders of education,' 
we hope to provide the support 
that teachers will need to meet the 
standards. 

In other words, after \00 years of 
observation and experimentation, this 
time we hope to get it right! ■ 

'The document was prepared by working 
group chairs Audrey Champagne. Henry 
Heikkinen, and Karen Worth. 

=For example. Association for Supervi- 
sion and Curriculum Development. Council 
of Chief State School Officers. Corporate 
Council for Mathematics and Science 
Education. National Parent Teachers 
Association, and National Governors' 
Association. 

'For example. Council on Foundations, 
National Council of State Legislatures, 
National Science Foundation, and the 
U.S. Congress. 
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The nature of elementary 
science: what does "it" ioolc iilce? 



These educators have now 
constructed and articulated 

an understanding of good 
science for their classrooms. 



by Gregg Humphrey 

IMAGINE A PARENT, or perhaps the superin- 
tendent, or maybe another teacher in your 
school, asking you the question: "What does 
the kind of science that you are doing look like? 
What is rr?" You might be tempted to reply by 
using words such as hands-on, activity-based, 
inquiry, or process; but the questioner asks: "If 
I were a child in your room, what might 1 be do- 
ing on a typical day?" 

These are questions that the participating 
teachers and principals of the Vermont Elemen- 
tary Science Project (VESP) began asking 
themselves last fall. These educators have now 
constructed and articulated an understanding of 
good science for their classrooms. Their Refer- 
ence Guide outlines what it is that students 
would know and be doing if they were engaged 
in hands-on, inquiry-based science. 

This profile of effective student practice is 
the result of teachers and administrators apply- 
ing the knowledge and methods of the Vermont 
Elementary Science Project, and then reflecting 
upon their classroom 
experiences with each 
other. This collabora- 
tive view of "ideal" 
student outcomes is 
particularly relevant 
to the development of 



new programs and to the assessment of inquiry- 
based, elementary science. 

As interest in the question of, "What does IT 
look like?" grew, we strategized ways for educa- 
tors to feel ownership in the process of 
developing profiles of effective science practice. 
The process used to produce these results is in- 
teresting in its own right. We asked teams to 
brainstorm ideas regarding good elementary sci- 
ence for the student. In other words, "What is 
IT," from the point of view of the child? 

During the next phase of developing a prac- 
tice profile, a committee met and attempted to 
categorize the data. Working in pairs, and then 
with the full committee, it was decided to use 
seven categories into which the ideas were sepa- 
rated (see page 9). Each committee member was 
then asked to write a short statement to capture 



the essence of a given categoiy. These state- 
ments were to describe "the ideal." 

At the next follow-up day of all VESP par- 
'icipants, the large group was separated into 
seven teams, one for each category of the "IT." 

Next, the teams viewed a videotape of 2nd 
graders investigating sinking and floating. We 
posed the question, "Is this IT?" Each group 
was charged with viewing the tape horn the 
point of view of the category that it was con- 
cerned with. Afterwards, each team shared with 
the large group "indicators" that were found 
from their particular category. More discussion 
took place refining the wording and ideas of the 
overall profile. 

Finally, the committee met again and pro- 
duced the work in progress known as "On the 
Run Reference Guide to the Nature of Ideal Ele- 
mentary Science for the Student." 

By increasing our ability to communicate 
about the "IT' with one another, we hope to be 
better able to plan the next steps: for the child, 
for the teacher, and for the school. If we con- 
tinue to develop a shared sense of the "IT," we 
all will be able to assess our progress towards 
defined goals. 

To use the analogy of a microscope with sev- 
eral objective lenses, there are different levels 
of magnification which help to define the "IT' 
for the student, the teacher, the school and, ulti- 
mately, the "IT' for the school system. We will 
delve into each of these layers as the Vermont 
Elementary Science Project continues its work 
with teachers. ^ 



GREGG HUMPHREY is ihe Technical Assistance Specialist 
for the Vermont Elementary Science Project (VESP). The 
VESP is a three-year grant awarded to the NETWORK, 
INC., Andover, MA, by the National Science Foundation. 
The ideas in this article were developed by teams of 
teachers and administrators from eleven schools in the 
Champlain Valley region of Vermont. It is their wish that 
this work be viewed as "in progress" and subject to 
revision. For more information, contact Maura Carlson, 
VESP Project Coordinator, Trinity College cf Vermont, 
McAuleyHall, Burlington, VT 05401 (802-65S-3664). 
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On the run reference guide 
to the nature of elementary science 
for the student 



■ ' To help answer the question: If students 
f an really "doing" hands-on inquiry based 
. ' science, what does it look like? 

- Children view themselves as 
" scientists in the process of 
' learning. 

' 1 . They look forward to doing science. 

2. They demonstrate a desire to learn 
■■' niore. 

3. They seek to collaborate and work 
cooperatively with their peers. 

4. They are confident in doing science; 
they demonstrate a willingness to modify 
ideas, take risks, and display healthy skepti- 
cism. 

Children accept an "invitation to 
learn" and readily engage in the 
exploration process. 

1 . Children exhibit curiosity and ponder 
observations. 

2. They move around selecting and using 
the materials they need. 

3. They take the opponunity and the time 
to "try out" their own ideas. 

Children plan and carry out 
investigations. 

1 . Children design a way to try out their 
ideas, not expecting to be lold what to do. 

2. They plan ways to verify, extend or dis- 
card ideas. 

3. They carry out investigations by: han- 
dling materials, observing, measuring, 
and recording data. 

Children communicate using a 
variety of methods. 
1 . Children express ideas in a variety of 
ways: journals, reponing out. drawing, 
graphing, chaning, etc. 



2. They listen, speak and write about sci- 
ence with parents, teachers and peers. 

3. They use the language of the processes 
of science. 

4. They communicate their level of un- 
derstanding of concepts that they have 
developed to date. 

Children propose explanations 
and solutions and build a store 
of concepts. 

1 . Children offer explanations from a 
"store" of previous knowledge. (Alterna- 
tive Frameworks, Gut Dynamics). 

2. They use investigations to satisfy their 
own questions. 

3. They sort out information and decide 
what is important. 

4. They are willing to revise explanations 
as they gain new knowledge. 

Children raise questions. 

1 . Children ask questions (verbally or 
through actions). 

2. They use questions to lead them to in- 
vestigations that generate funher 
questions or ideas. 

3. Children value and enjoy asking ques- 
tions as an imponont part of science. 

Children use observation. 

1 . Children observe, as opposed to just 
looking. 

2. They see details, they detect sequences 
and events; they notice change, similari- 
ties and differences, etc. 

3. They make connections to previously 
held ideas. ^ 



Work in progrest Vermont Elementary Science Project, 
Trinity College. McAulex Hall. Burlington, VT 05401 
(802)658.3664. 
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On The Run 
Reference Guide To 
The Nature of Elementary Sdence for the Student 

To help answer the question: ^ students are really 'doing' hands-on inquiry based science, what does it look like? 

Children View Themselves as Scientists in the Process of Learning. 

1. Ihey look fbiwaid to doing science. 

2. Ihey demonstrate a desire to learn more. 

3. The seek <o collaborate and work cooperatively with dteir peers. 

4. They are confident in doing sdence; ttiey demonstrate a willingness to modify ideas, take risks, 
and display healttiy skepticism. 

Children Accept an Invitation to Learn" and Readily Engage in The Exploration Process. 

1. Qiildren exhibit curiosity and ponder observations. 

2. Ihey move around selecting and using the materials they need. 

3. Ihey take tfie opportunity and the time to "try out" tiieir ovm ideas. 

Children Plan and Carry Out Investigations. 

1. Children design a way to try out their ideas, not expecting to be told what to do. 

2. Ihey plan ways to verify, extend or discard ideas. 

3. Ihey cany out investigations by: handling materials, observing, measuring, and recording data. 
Children Communication Using a Variety of Methods. 

1. Children express ideas in a variety of ways: journals, reporting out, drawing, graphing, charting, 
etc 

2. They listen, speak and write about science witti parents, teachers and peers. 

3. They use the language of processes of science. 

4. They communicate tiieir level of understanding of concepts diat dtey have developed to date. 

Children Propose Explanations and Solutions and Btiild a Store of Concepts. 

1. Children offer explanations from a "store" of previous knowledge. (Alternative Frameworks, Gut 
Dynamics). 

2. They use investigations to satisfy their own questions. 

3. They sort out information and dedde what is important 

4. They are veiling to revise explanations as they gain new knowledge. 

Children Raise Questions. 

1. Children ask questions (verbally or through actions.) 

2. They use questions to lead them to investigations ttiat generate further questions or ideas. 

3. Children value and enjoy asking questions as an important part of sdence. 

Children Use Observation. 

1. Children observe, as opposed to just looking. 

2. Ihey see details, they detect sequences and events; they notice change, similarities and differences, 
etc. 

3. They make connections to previously held ideas. 

Children Critique Their Sdence Practices. 

1. They use indicators to assess their own work. 

2. Ihey report their strengths and weaknesses. 

3. They reflect with their peers. 

01/25/91 VESP Practice Profile: The Student "if. 
04/92 Revised 

The Vermont Elementary Science Project is located at Trinity College, McAuley Hidl, 
Burlington, VT 05401,(802)658-3664. 
The VESP is a grant awarded to The NETWORK Inc., Andaoer, MA, by the National Science Foundation. 
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% Assessment: what is "IT?" 



by Gregg Humphrey 

"THE ANSWER is not to be found in books 
aboui tests and examinations, but in what is 
happening in classrooms every day." This ob- 
servation by the well-known British researcher, 
Wynne Harlen, is central to our approach to 
assessment at the Vermont Elementary Science 
Project. It is essential that educators develop 
assessment strategies that are effective and that 
do not force teachers into unworkable modes of 
teaching. 

The term "assessment" is applied to a wide 
range of methods by which information about 
students is gathered and appraised, including 
formal testing and analysis. Gathering informa- 
tion about children as a result of day-to-day, 
informal interactions is part of teaching and is 
equally a part of assessment. 

For many teachers, administrators, and par- 
ents the word "assessment" brings to mind 



facilitate the activities as well as to assess con- 
ceptual, process, and attitudinal indicators 
congruently. 

A fundamental piece of assessment occurs as 
teachers and students are engaged in the daily 
activities of hands-on, inquiry-based science. 
Just as teachers listen to students read and diag- 
nostically raise questions and suggest strategies 
for improvement, teachers interact with children 
as they "do science" in order to determine their 
progress and plan for future instruction. 

Assessment to benefit learning 

We need to develop and adopt assessment tech- 
niques that match what is valued in science 
learning and that reinforce the goals of science 
education. For example, rather than just testing 
for factual recall, we should assess the extent to 
which students can apply the knowledge they 
possess. We must develop techniques to probe 
for depth of understanding of facts 



A fundamental piece of assessment ^"'^ ^°""p^. ^"'^ '^'^ ''^^'^ '° 

, 1 ^ ^ ^ problem solvmg skills 

occurs as teachers and students are 



engaged in the daily activities of 
hands-on, inquiry-based science. 



"testing." It is important to broaden and change 
this limited conception of assessment to a wider 
view which promotes the growth of science and 
math concepts, process skills, and attitudes that 
are inherent in effective science and math devel- 
opment. 

A common misunderstanding is that assess- 
ment is something that requires additional 
and/or different activities — as an adjunct to 
teaching. To some it means that teachers have to 
stop teaching and withdraw from interaction 
with children in order to observe them. Yet with 
effective reading and writing instruction, teach- 
ers routinely make immediate assessments 
based on behaviors and abilities which students 
exhibit in the act of reading and writing. These 
judgments, or "continuous assessments." often 
result in probing questions and other pedagogi- 
cal decisions on the part of the teacher. Such 
judgments are used in the service of instruction 
as well as to monitor and record both short and 
long term student development. Similarly, when 
students are doing science, teachers can learn to 



As classrooms increasingly use 
hands-on approaches for building 
students' understanding of scien- 
tific concepts, hands-on 

performance can be used to assess 

student competencies and skills. Further, we 
need to discard the view that assessment is only 
done after completion of a unit and for the pur- 
pose of giving students a grade. Assessment 
needs to be a means for improving instruction, 
not just measuring its impact. Day-to-day 
assessment can serve instruction by helping 
teachers monitor the progress of students in sci- 
ence so that adjustments to enhance student 
learning can take place. ^ 



GREGG HUMPHREY is the Technical Assistance Specialist 
for the Vermont Elementary Science Project. His article, 
"The Nature of Elementary Science: What Dots "IT" Look 
Like?'' appeared in the September 1991 issue of Connect 



"Assessment in 
education has been 

criticized for 
interfering with the 
process of learning, 
the analogy being 
that of the gardener 
constantly pulling 
up his plants to see 

if the roots are 
growing. There is 
some truth in this 

... but it also 
distorts reality to 

make a point. 
Gardeners do have 
to fmd out if their 
plants arc growing 
and they do this, 
not by uprooting 

them, but by 
careful obsen'ation 

with a 
knowledgeable eye, 
so that they can 
give them water 
and food at the 
right times and 
avoid either 
undemounshment 
or over-watering 
. . . The gardener 
who does not knov. 

what size and 
shape a plant is to 
be and how quickly 

it is expected to 
grow will not know 
what signs to look 

for. and may 
mistake a condition 
which is quite 
normal for one 
which requires 
remedial action, or 
vice versa. We have 
to know something 

■bout the 
development taking 
place in order to 
imeipret what we 
find when we 
assess it" 
(p. vii) 

Sdinet: Cuidtt to 
Atttssmentin 
EiiuMion, 1983. 
V^TUie Harlen 
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What's Worth Assessing? ' 



BY MONTE MOSES 
Associate Superintendent. Cherry Creek Schools. Englewood. Colorado 



Over the past several years I have 
been invoh^ed in alternative as- 
sessment projects as a district 
leader, principal, and consultant These 
experiences have convinced me that 
high quality assessments can enhance 
student growth and achievement. 

Along the way, I have discovered a 
few ideas that odiers might consider 
and \jsc. 

First, get clear on what alternative as- 
sessment really is. I advise colleagues to 
focus their energies on creating 'perfor- 
mance assessments'* as opposed to "al- 
ternauve assessments" because the 
words indicate better the kind of mea- 
sures that are needed in today's schools. 
Performance assessments focus on spe- 
cific results, while alternative assess- 
ments could be virtually anything. 

Design tasks that are worth assess- 
ing and that provide useful informa- 
tion on core district outcomes. Al- 
though it sounds easy, not enough 
rime is spent defining the assessment 
task. The tendency is to look for 
things easy to measure instead of ask- 
ing what is important to assess. 

I encourage administrators and 
teachers to begin with this quesuon, 
"What task could students perform 
that would be a strong predictor of 
later success in school and life?" 
These are the things we should teach, 
not jusc assess! 

At Mark Twain Elementary School 
in Littleton, Colo., we reached agree- 
ment that the ability to research a 
topic of social or scienufic impor- 
tance is one such task. We viewed diis 
task as very important because it calls 
for students to use skills such as read- 
ing and writing and to integrate 
knowledge from different subjects. 
Better yet, it is used repeatedly in all 
levels of schooling and walks of life. 

We envisioned the research assess- 
ment as being much more in-depth 
than simply wriung a report, which 
our students had done a great deal 



but with questionable quality. Our 
goal was to design an assessment task 
and scoring system that would get stu- 
dents to move beyond the plagiaristic 
tendencies so typical of elementary 
school children. 




jneasurej 



We discovered the "trick" to good as- 
sessment isn't getting students to do 
something completely foreign. Rather, 
it is a matter of being clear about what 
quality means. 

Linking Instruction 
The assessment tasks themselves must 
be engaging. Students must be inter- 
ested in an assessment task before we 
can assume it is their best work, and 
then hold them accountable to high 
standards of thoughtfulness and self- 
direcrion. Thoughtfulness and self-di- 
recrion demonstrate the student pos- 
sesses more than just knowlet^e and 
skills. They show the student has de- 
veloped the atutudinal disposition 
needed to apply them. 

Administrators must help teachers 
develop a tight conceptual linkage 
among curriculum, instruction, and as- 
sessment. Without this linkage, teach- 
ers are likely to view performance as- 
sessment as an intnision into instruc- 
tional time and a diversion irom "cover- 
ing" the curriculum. The more produc- 



tive mindset is to see performance as- 
sessments as "tests worth teaching to." 

As performance assessments are 
developed, make sure everyone un- 
derstands they exist to strengthen 
teacher judgment, not replace it. The 
information obtained from a perfor- 
mance assessment can be quite help- 
ful, but it is still up to the teacher to 
interpret and me this information. 

Anotiier important step is to get stu- 
dents involved in self-assessment of 
their performance against predefined 
standards. Wouldn't it be wonderful if 
students could appraise their own 
work against standards in a manner 
identical to what the teacher would 
do? This idea is not at all far fetched. 

Start now to create performance as- 
sessments. Personal experience is the 
key in this endeavor. Until teachers 
have struggled with the problems of 
task selection and setting standards, 
they will not fully appreciate the in- 
structional value of good assessment. 
The more staff who are involved in 
these discussions, the better. Involv- 
ing parents and commimity members 
is also a good idea. They can add a lot. 

Don't be afraid to make revisions 
based on feedback from parents, 
teachers, and students. The research 
assessment at Mark Twain Elementary 
was revised substantially seven times. 
We had to swallow our pride, but die 
end result was worth it. 

After working out the kinks on the 
research performance assessment, we 
institutionalized it. The assessment 
was given to students several times 
during tiieir elementary school expe- 
rience, in hopes they would improve 
from one try to die next. The assess- 
ment took on die character of the 
big game" and motivated smdents to 
prepare well in advance. Better per- 
formance was the result. 

Altering Perceptions 

Among many obstacles, two stand 
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out The fint is resources. The devel- 
opment of performance assessments 
takes time, and time costs money. 
However, the initial funds required 
are small in comparison with what dis- 
aicts commonly spend on commer* 
cial testing, whose value often is ques- 
tioned. Also, keep in mind that the 
investment in performance assess- 
ment is good staff development 

The second obstacle relates to 
adult perceptions of students. One 
perception can be expressed in the 
form of a question, "What will we do 
with the students who fail the perfor- 
mance assessment?" This question as- 
sumes some students cannot learn, 
grow, and achieve when expectations 
are high. School leaders must con- 
front that dangerous assimiption. 

Tiie question also fiatils to consider 
performance assessments are not 
one-shot tests. They are tasks worth 
repeating regularly because of their 
inherent value. Our job as educators 
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is to coach students to higher perfor- 
mance from one try to the next 

Another obstacle concerns able stu- 
dents. The highest score on a perfor- 
mance assessment should be set be- 
yond the current skill level of even the 
best students, an idea which will seem 
quite foreign to them. Many of these 
able students (and their parents) in- 
correcdy assume that their work is ex- 
cellent because it is better than theii 
peers, when it may be far below stan- 
dards of excellence. Strong students 
deserve a challenge that will call upon 



Students partidpating m Ntw York state's 
/(mrthgmdesdmce manipulative skills 
assessment are asked to miasuTt objects using 
a ruler, double-pan balance, or a thermometer. 

them to grow and stretch to meet 
high standards Just as much as an av- 
enge or below-average student 

Performance assessments give us a 
unique opportunity to raise standards 
a nd he lp more students to reach 
iem tfiaiT-«v5r before. In my mind, 
that is worth puhimig. 
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Creating Benchmarks 
For Science Education 



Andrew Ahlgren 



Progression-of- 
understanding 
maps were one 
important tool 
in coordinating 
tlie research 
and information 
needed for 
creating K-12 
science 
standards. 
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reject 2061 has been consmicting 
goals for science, mathematics, and 
technology education since 1985. 
During our first three years of work, 
we recommended what students 
should remember by the time they leave 
high school (Science for All Americans 
1989). Since 1988. we've been working on 
reasonable expectations for students at 
earlier grade levels (Benchmarks for 
Science Literacy, in draft). This new 
volume will include benchmark lists, some 
of our "progression-of-understanding 
maps." and essays related to the bench- 
mark topics. 

We intend the benchmarks to be used by 
school districts or curriculum developers 
in constructing alternative K-12 curriculum models 
adapted to their own populations and circum- 
stances. Before we reach that point, though, we 
believe some reflection on how we created the 
benchmarks can be a sumulus to other curriculum 
reform efforts. 

The experience of writing benchmarks is highly 
stimulating. But make no mistake about it: The 
work is difficult, and getting started seems to 
discourage many people from the undertaking. Still 
the quality of thinking and conversauon that goes 
on is often impressive, even when the tentative 
product may not be. 

Benchmark Grades 

The National Assessment of Educational Progress 
(NAEP) has popularized grades 4 and 8 as bench- 
mark grades, and the National Council of Teachers 
of Mathematics (NCTM) has followed that pattern 
(Curriculum and Evaluation Standards for School 
Mathematics 1989). However, the district teams 
working with Project 2061 decided that the 
psychoioeical distance from K to 8 requires more 
than a single benchmark. The end of grades 2 and S 
were recommended as being more meaningful 
developmental breaks, and we are designing 
benchmarks for those grades. 

It is not our intention that 2nd graders should be 
subjected to formal, national examinations on their 



progress in science (as seems to be in the cards for 
older children). The "feel" of the expectauons for 
grade 2 is disunctly different from that for grade 5, 
so we believe it is important to discourage kinder- 
ganen teachers from embarking immediately 
toward grade 5 goals. By their modest, nontech- 
nical nature, grade 2 benchmarks suggest what 
students cannot be counted on to know as they 
begin grade 3, and this may temper expectations 
for what children can learn in grades 3 through 5. 

Inferring Benchmarks 

In crafting t'le lower-grade expectations, we drew 
partly on an analysis of what ideas would be 
needed to achieve the !2th-grade understandings in 
Science for All Americans. We also considered esti- 
mates of what students are capable of at different 
ages, drawing information from the experienced 
teachers on our district teams and from researchers 
who study how children understand and learn 
science. Unfortunately, the availability of 
published research on children's understanding of 
science is very uneven over different content areas. 

We have found that it is seldom possible to 
work backward from 12th grade goals one at a 
time to create a neat stack of previous levels of 
sophistication. Usually there are convergences 
(several ideas required to understand a subsequent 
idea) and divergences (several ideas depending on 
one prior idea). The natural medium to express 
such goals is therefore a diagram with boxes and 
arrows. We called our speculative charting a 
"progression-of-understanding map" (to distin- 
guish it from the "concept map" currentiy popular 
in science education). 

Figure 1 is an example of a draft of a progtes- 
sion-of-understanding map, depicting ideas related 
to a section on the structure of matter (Science for 
All Americans. Chapter 4: "The Physical Setting"). 
Reading from bonom to top, the map shows a 
rough progression in time, beginning from nouons 
students hold when they enter school. When we 
sketched this map, the sequence of ideas was more 
important thaii tying ideas to any particular grade. 
Estimation of approximate grade placement for 
each idea usually came later. 
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Progression-of-Understanding Map 
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Unfortunately, 
the availabili.ty of 
published resv^arch 
on children's 
understanding of 
science is very 
uneven over 
different content 
areas. 



The toughest part of the map was in the middle. 
Without the finn gu-dance of research on many 
topics, we were in the same place as the Geography 
Task Force of the National Council on Education 
Standards and Testing when they wrote: "It was 
difficult to set 8th grade standards, other than indi- 
cating that students should be expected to know 
more than they did in the 4th grade and less than 
the 12th grade" {Raising Standards for American 
Education 1992, p. L-4). Compounding our uncer- 
tainty was the possible difference between what 
children could be expected to do now, with their 
current histoty in the school system, and what 5th 
or 8th graders eventually might be able to do if 
they had optimal experiences. 

When we began mapping, we intended to cover 
one conceptual strand at a time, leaviug some loose 
ends that would later connect to other strands. For 
example, the structure of matter shows obvious 
connections to the flow of matter and energy. The 
structure of matter was a whole section in Science 
for All Americans, and perhaps it was too large a 
concepmal chunk to represent comfortably on a 
single map. (Notice that it is incomplete in fig. 1.) 
It soon became evident that a progression-of- 
understanding map for a single strand was already 



more complex than most people would fmd 
inviting. 

Software support for constructing each map and 
for making connections among them would be very 
helpful (and we now have a grant to develop such 
software). We intend to create a curriculum 
resource data base that will link appropriate parts 
of the progression-of-understanding maps, giving 
users the option of choosing the complexity of 
information they want to consider. The resource 
base would also link benchmarks to blocks, to 
activities and materials, and to appropriate assess- 
ment suggestions. ■ 

We plan to accompany benchmark lists with 
essays that will call attention to the progressions of 
various strands and connections among them. For 
example, the essay accompanying the benchmarks 
for the structure of matter would draw attention to 
the parallel development of four different strands: 
properties of substances, combinations of parts, 
invisibly small pieces, and conservation of matter. 

Es$ays, which are being prepared by our teams 
and staff, will also draw on the available educa- 
tional and psychological research, calling attention 
to difficulties that students are likely to have at 
each level and, in particular, to persistent previous 
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Literacy eoak The Stractura of Matter 

The following example of an essay and 
benchmark list is taken ^om me draft of 
Benctvnarks tor Scianc* Literacy (in 
draft). 

Students will learn about the nature of 
atoms and molecules and the stnjcture 
of matter. 

Of all sections, this one may have the 
most implications for students' eventual 
understanding of the picture that 
science paints of how the world works. 
However, it may also offer the most diffi- 
culties. The theory of atoms and 
molecules is powerful in explaining our 
world, but it requires bringing together a 
number of lines of evidence and imagi- 
nation: about the properties of materials 
and their combinations, changes of 
state, effects of temperature, behavior 
of large collections of pieces, the 
construction of objects from parts — 
even at^out the desirability of simplicity 
in explanation. All of ihese should be 
grasped by children during middle 
school, so that the unifying ideas of 
atoms can be developed by the end of 
grade 8. 

The scientific understanding of atoms 
and nriolecules requires students to 
entertain the notion that all visible things 
are composed of invisible particles. 
Another notion is that everything might 
be made up of a relatively few ingredi- 
ents. An idea preliminary to this is that 
materials combined in different ways 
can have different prope-rties. And still 
preliminary to that is the very notion of 
■properties" of materials. 

Parallel to consideration of properties 
of combinations is the notion that the 
bulk properties of materials can be very 
different from the properties of their 
minute parts — an idea counter to the 
students' intuition.' And parallel too is 
the idea of an unchanging total amount 
of matter, beginning with the evidence 
that total weight stays the same during 
all sorts of cnanges in materials.' 

6rades 3 through 5 

The study of materials should continue 
throughout these years but become 
more systematic and quantitative. 
Students should design and build things 
that put different requirements on the 
properties of materials. They should be 
expected to write clear descriptions of 
their designs and experiments, present 
their findings whenever possible in 
tables and graphs (designed by the 



students, not the teacher) and enter 
their data and results in a computer 
database. 

Students should measure (weight, 
dimensions, temperature), estimate 
(dimensions, weight population size), 
and calcutate (area, volume, population 
size) using hand-held calculators when 
necessary.* With magnifiers, they should 
inspect substances composed of large 
collectiotis of particles — sand, spices, 
powders— to discover the unexpected 
details at smaller scales. They should 
observe and describe the (sometimes 
solid-like, sometimes liquid-like) 
behavior of large populations of 
pieces— powders, marbles, sugar 
cubes, or wooden bkjcks. 

By the end of the 5th grade, students 
should know ttiat 

■ Heating and cooling cause 
changes in the properties of materials. 
Many kinds of changes occur faster 
under hotter conditions. 

■ However parts are assembled, the 
weight of the thing made is always the 
same as the sum of the parts; and when 
a thing is broken into parts, the parts 
together weighed ttie same as the orig- 
inal thing. 

■ Materials may be composed of 
parts that are too small to be seen 
without magnification. 

■ When a new material is made by 
combining two or nxxe other materials, 
it can tiave properties that are different 
from any o^them. For that reason, a k5t 
of different kinds of materials can be 
made from a small number of basic 
kinds. 

■ A collection of a large number of 
pieces may keep its shape or flow like a 
liquid, depending on how the pieces 
stick together or how they are stacked. 

'Brook et al. 1964. Driver 1987. 

'At the Mglnnmg, children have varnus 
ideas atxxjt what is 'matter.' For many, gases 
and even liquids are not seen at having weight 
or as being matter (Lee ei al. in press. Driver 
1987. Stavy 1990). Very tmy solid particles are 
also not seen as having weight— because their 
weight cannot be telt (Smiih. Carey, and Wiser 
1965. Carey 1991).. 

"Research shows that children may consider 
anything so light that they cannot feel its weight 
to have no weight at all (Smith, Carey, and 
Wiser 1965: Carey 1991). Lots of weighing on 
increasingly sensitive balances, including 
weighing piles of small things and dividing 
them to find the weight of each, will help. 



conceptiotis that may interfere with 
learning (see fig. 2). We are still 
uncertain about now far essays should 
go beyond suggesting appropriate 
kinds of instruction. (The research has 
much less to say about instruction to 
overcome difficulties than about the 
difficulties themselves.) 

BenclimarkAdiustinents 

Once in the thick of producing bench- 
marks, occasions will arise when a 
benchmark statement doesn't seem 
well suited to its designated grade 
level. The easiest option is to move the 
benchmaric intact to another grade 
level, making adjustments to any 
benchmarks connected to it. 

A second option is to rewrite the 
statement at a level of sophistication 
more appropriate to the current grade 
level, but this is more than a stylistic 
transformation. Grade adjustments can 
seldom be made so simply as 
changing the vocabulary. Rewriting 
usually requires rethinking what 
students should be able to do— in their 
heads or behaviorally — at that level. 

A third rewriting option is the most 
difficult but probably the most fruitful: 
tease apart the substance of the bench- 
mark and create two new ones, 
keeping one at the current level and 
putting the other at a different one. 
Again, merely making style changes 
in language won't change the 
substance of the benchmark. One must 
reconsider what students could under- 
stand and what the likely sequence of 
understanding is. 

Knowledge vs. Belief 

Research shows that children may 
understand a scientific explanation of 
phenomena before they believe it (for 
example, Hewson and Hewson 1992, 
Osborne and Freyberg 1985). The 
longer the time gap between being 
able to state an idea and eventually 
believing it. the greater the problem 
for writing benchmarks. Should a 
benchmark about children's ability to 
explain something specify that they 
can produce a scientific explanation, 
or should we try to require their accep- 
tance of it as well? 
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From a philosophical point of view, 
Project 2061 would prefer to require 
knowledge rather than belief. A 
similar dilemma appeared in writing 
the "Values and Attitudes" section in 
Science for All Americans. We 
rejected the goal that everyone should 
like science, mathematics, and tech- 
nology or should believe these 
endeavors are of net benefit to 
humankind We agreed instead on the 
goal that students' attitudes — ^whether 
they turn out to be positive, negative, 
or neutral — should be based on a 
sound understanding, 

A poignant case might be that of 
evolution through natural selection. 
We can reasonably require that 
students understand what the scientific 
theory is, but do not have to require 
students to believe that is how present 
life on earth necessarily came to be. 

Final Thoughts 

Benchmark drafts should be ched out 



with a variety of readers, not just for 
approval or minor editing, but to see 
how they are likely to be interpreted 
and used. Writing good benchmarks 
may not require setting fixed rules as 
much as it requires being continually 
vigilant about how one's intent might 
be misunderstood. 

Researcher Pat Heller (of the Univer- 
sity of Minnesota) summarized the task 
for us after a recent writing retreat: 

■ Make benchmarks not so specific 
as to be limiting and not so general 
that no one is quite sure what you're 
talking about 

■ Have a dear sequence where 
necessary within a grade level. 

■ Have a progression from one 
grade level to the next that illustrates 
increasing sophistication. 

■ Show connections between 
benchmarks under different goals. 

■ Write them to be developmentaily 
appropriate, assessable, and relevant 
to the child's world. ■ 
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Assessment, 
Practically Speaking 



How can we measure hands-on science skills? 

By Lehman W. Barnes and Marianne B. Barnes 



THOUSANDS OF ELEMEN- 
taty school children busily 
do handson science ac- 
tivities even' day. They 
observe and classify ob- 
jects, measure length, 
volume, and mass, collect and record 
data, and manipulate materials. These 
students know that science involves 
both doing and communicating. But 
as they prepare for science tests, what 
have students come to expect? Are 
they being assessed on all that they've 
learned? 

The Motor-and-Wires Kid 

Jimmy was thrilled when we began a 
science unit on electricity and mag- 
netism. I had assigned the firet 11 
pages of chapter four and was begin- 
ning to discuss the main ideas of the 
unit. Suddenly Jimmy buret out, "1 
have a couple of motors at home, could 
I bring them into class and show them 
to everyone? Or one of the 'electric 
magnets' 1 made last summer? I saw a 
picture of one in a book about science 
at the grocery store. All you need are 
some wires, a nail, and a battery." 

"Sure Jimmy," I said, glad to see 
him excited. "Bring in your motor; 
and wires." 

The next day, as I was unlocking 



Doing a hands-on biology activity. 

the class door, I heard the rush of 
footsteps. There was Jimmy with a 
large paper bag containing all kinds of 
stuff— batteries, motors, wires, nails, 
pliers, and tape. "When are we having 
science today?" he asked me. His ex- 
citement was evident as he pored 
through his bag. 

Teachers thrive on flexibility, espe- 
cially in response to student enthusi- 
asm, so I began science right after the 
morning announcements, a time usu- 
ally reserved for language arts. Jimmy 
showed his classmates the motors. 



batteries, and wires, demonstrated how 
to build an electric magnet, and built 
an electric circuit He eagerly shared 
the contents of his bag with the other 
students, most of whom-were equally 
eager to learn what he knew about the 
motors. 

During the next two days, with the 
class's interest and exdtement levels 
staying high, I introduced the main 
ideas of the science unit and, thanks 
to Jimmy, got the whde class involved 
in creating series and parallel circuits. 
I ended the actmties with a brief review 
for Friday's science test 

What Does My Test Test? 

The comprehensive test— vocabulary, 
labeling, matching, multiple-choice, 
true-or-felsc, short-answer, and puzzle 
questions— accurately assesses stu- 
dents' mastery of the verbal aspects of 
the science unit. Jimmy did well that 
Friday, especially on the short-answer 
questions. His answers were complete, 
and he even included drawings of his 
two setups. At the bottom of his test 
paper he wrote, "1 think I know a 
whole lot more, but I can't show you 
on this kind of test" 

Jimmy's comment struck me. The 
test lacked an opportunity for students 
to demonstrate what they had learned 
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of science would be better evaluated ; 
in such a manner. For example, think | 
about evaluating science skills in the | 
following scenarios: 'Z 

• working with basic science equip- | 
ment,suchasathemiometer,atriple ^ 
beam balance, a meterstick, a grad- 1 
uated cylinder, and a stopwatch; 

• perfonning laboratory skills and pro- 
cedures, such as working with a 
magnifying glass or microscope, 
heating or filtering a substance, 
mixing a solution, or measuring 
rates; 

• observing and classifying three- 
dimensional objects, such as shells, 
rocks, plants, and animals (currently, 
most assessment tests employ visual 
observation only in two^limensional 
settings); 

• collecting and recording data in 
tables, charts, and graphs that stu- 
dents create themselves— such 
graphs could reflect temperature 
changes over time, heartbeats per 
minute, or a ratio of the manipulated 
variable to a responding variable: 

• designing experiments and perform- 
ing investigations from a set of 
materials and a specific question 
(Which paper towel will hold the 
most water? How can you make the 
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Getting 
Connected to 
Science 



by Candace L. Julyan 



Dr. Candace L Julyan is Project Director of the 
NGS Kids Network and Principal Investigator/ 
Project Director of NGS Kids NetworkI Middle 
Grades. 



Consider how you Sxst became interested in science. Perhaps yoiu- 
interest grew from a curiosity about a paitiotlar phenomenon or organism. 
Satisfying that curiosity, or what IDavid Hawkins (1967) calls "messing 
about," often resulted in some type of relatively vmstructured exploration led 
by your own questions rather than the questions of others. Unfortunately, 
messing about is not a common practice in many science classrooms today. 
Students are more likely to be introduced to organisms and phenomena 
through text-based lectures rather than through their own observations. The 
result of our current science curricula is not only a lack of interest in science as 
a profession but also a lack of scientific tmderstanding. 

Data from numerous studies (such as Bell and Brook, 1984; Ehickworth et 
aL, 1985; Julyan, 1988) suggest that knowing correct terms does not help 
students make sense of their observations. In fact, in many cases scientific 
terms may be used to mask confusion. 

Although I am certainly not suggesting a classroom ban on scientific 
terms, I do suggest that we reconsider their importance. The difference 
between memorizing vocabulary words related to science topics and imder- 
standing various phenomeiui is considerable. CertaiiUy those of xis involved 
in science and science education realize the difference and are striving for the 
latter. Sdence-as-vocabulary requires less effort on the part of both the 
teacher and student, but it also provides fewer rewards. 

Students are also aware of the difference and, if given the choice, they too 
would strive for imderstanding. This point was highlighted for me several 
years ago by a high school student who was participating in a research study 
on how students make sense of a complex system (Duckworth et aL, 1985). To 
examine this research question, my colleagues and I had devised a number of 
experiments featuring helium balloons weighted with enough strands of 
string so that the string dragged on the floor. The students' task was to 
explain why a balloon's position in the air changed as they did various tasks 
such as tying knots in the string, cutting the string, and putting the balloon on 
a table. 

Early in the study, one student nuide it de&r that she would not accept a 
scientific term as an explanation for what she was seeing. Several students 
consistently used the word "gravity" to explain anything that they could not 
easily make sense of otherwise. This particular student protested that expla* 
iiation vehemently, exclaiming that she had never understood what gravity 
was and that the word certamly did not help her now as she worked with the 
balloons. 

After several weeks of experimenting, the balloons continued to surprise 
her, and her frustration at her lack of understanding was often visible. One 
day, she turned to me demanding to know if I intended to tell her "the 
answei." When I asked her to state the question, she seemed momentarily 
stumped but then stated that she wanted to know why the balloon behaved 
in the way that it did in all the various experiments she had conducted. 
Although certairUy willing to answer her question, I said that I wanted to be 
dear about what she wanted. Did she want words that explained the 
phenomenon or did she want to understand it herself? There was a long and 
poignant silence in the room. Finally, she turned and quietly said that the 
words were not what she wanted, she really wanted to understand. 

Although most students might not articulate the dilenuna as dearly as this 
student, many share her desire to understand, not just to know, the correct 
words. One way to promote both understanding and the value of sdentific 
terms is to give students an opportunity to mess about in their sdence classes, 
to become partidpants in constructing their knowledge. Inquiry-based 
activities are certsiinly not the fastest way to approach sdence learning. The 
faster and more effident approach is definitely text-based, lecture-based 
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dasses. However, this more common 
approach does not appear to be ter- 
ribly effective. Students' notions about 
sdentilic topics often differ from those 
of scientists and form a coherent fabric 
of understanding that has its own in- 
ternal logic A presentation of new, 
sometimes corilicting, information by 
a teacher does itot necessarily change, 
or even challenge, students' initial 
ideas about the topic This is an impor- 
tant point to keep in mind — students 
do not learn simply by being told. 

One of TERC's goals through the 
years has been to encourage educators 
to consider the possibilities and the 
strengths of "messing about" in science 
and mathematics. For the past several 
years we have taken a new perspective 
on student inquiry through the 
projects we call "network science." The 
basic premise of these curricula is that 
students can and should be scientists, 
that they can and should converse with 
real scientists about their work, and 
that computers can enhance this 



from Natioiud Geographic IQds Net- 
work to Star Schools to Global Lab. 
Although each program has unique 
characteristics, as a group they repre- 
sent an exploration into the power of a 
particular way of nessing about Most 
of the tmits in these supplemental 
curricula focus on an environmentally- 
oriented topic such as add rain, trash, 
water quality, or radon. Students 
usually begin their study by examining 
the topic in the context of their local 
community and then expand their 
inquiry by exploring it within the larger 
national and intematioiud picture. 
Students collect some type of data 
(survey or measurement), share those 
data through telecommunications with 
other dasses collecting the same data, 
and finally make sense of those data 
using computer (and non-computer) 
tools. As much as possible, we have a 
sdentist, a professional with expertise in 
the unit topic, available to the students 
to hdp them consider the questions 
raised by the data. 




TERC Star Schools students working intently on their group project. 



enterprise. Students conduct experi- 
ments, analyze data, and share results 
with their colleagues using a simple 
computer-based telecommunications 
network. This collecting and making 
sense of data gives the students an 
opportunity to experience the exdte 
ment of sdence that sdentists fed. 
Our network projects are diverse, 



We view the computer as an impor- 
tant tool in these inquiries, providing 
students with help in both data collec- 
tion and analysis. The typical computer 
configuration features a word proces- 
sor, a record-keeping data section, a 
graphing utility, a complete telecom- 
munications package, and map soft- 
ware with data overlay. 



Our network sdence projects have 
generated an enthusiastic response on 
the part of both teachers and stiidents. 
One NGS Kids Network teacher 
explained network sdence as "an 
aw^me concept, a truly revolutioiuSPlF 
idea for education at a time when it is so 
badly needed." At present we have 
tested these materials in approximatdy 
5,000 classrooms across the United 
States and at a growing number of 
fordgn sites, induding Argentina, 
Australia, Gmada, Hoi^ Kong, Israel, 
the Soviet Union, and 2^inbabwe. Data 
£rom our various evaluations suggest 
that network sdence is a lively and 
appealing v«iy to approach sdence. 
From these data we have come to have 
a nimUser of "beliefs" about the possi- 
bilities of this approach. We offer them 
as points for consideration and discus- 
sion for a wide audience of educators, 
£rom teachers to administrators to other 
developers. 

Data collection and analysis 
provide an effective backbone 
aiound which to study science. 

In addition to the expected leamingg^ 
about the particular subject covered iiVP 
the tmits, teachers and students re- 
ported numerous examples of sdentific 
thinking generated by the simple act of 
collecting and making sense of data. In 
many ii\stances, units begin by asking 
students to consider definitions, not as 
vocabulary exercises but as an impor- 
tant aspect of understanding data. 
Students in one fifth-grade dass, 
preparing to collect data on trash, spent 
considerable time dedding whether 
"used" was different from "you don't 
need it anymore." The longer students 
discussed the difference, the more they 
realized the connection between trash 
and people's personal habits. This class 
finally agreed that trash was "some- 
thing that has used its purpose." 

TTiis definition matched the thiiJdng 
of some other dasses that defined trash 
as "something no longer useful to a 
person" or "useless, worthless, worn 
out, imwanted, and of no value to you/V 
Other classes' discussions about trash^fk 
seemed to look at trash in a slightly 
different way as they considered trash 
in relation to waste reduction solutiora. 
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These students defined trash as "any 
object that a person doesn't want or has 
no further use for, or that is not being 
vded. reclaimed, repaired, or 
^ded or that may pollute the 
feuth" or "waste, scraps, garbage, litter, 
old things that are no longer in use, 
spoiled and ruined things, and things 
that can be recycled but weren't" As 
students continued the unit and shared 
data about the actual trash collected, 
they also considered the ways in which 
students' and communities' ideas 
about trash might affect the actual data. 
The exerdse of considering all the 
complications of data collection and 
analysis is a vital step in the students' 
understanding of science. 

Another example of the type of 
scientific imderstanding that data 
collection generated came from a 
fourth-grade class. These students 
reailized that although their data and 
those of their teammate school were 
both about pets, they could not com- 




A hards^n activity of the NGS Kids 
Network. 



pare data. One class had collected the 
number of students who ovmed each 
i of pet; the other had collected the 
nber of each kind of pet. The class 
earned a great deal about the impor- 
tance of comparable data, a concept 



that many high school students might 
find difficult 

Technology greatly enhances 
classroom inquiries. 

In many ways technology can pro- 
vide an important link between science 
in practice and science in classrooms. 
Through the power of telecommimica- 
tions, students are motivated to 
collaborate and share data and ideas 
with other students from all around the 
world. By using microcomputers for 
computations, students are able to 
maniptiiate their findings and explore 
their ideas in more sophisticated ways 
than their computational skills might 
have permitted otherwise. And 
telecommunications offers a unique 
and manageable opportunity for 
scientists to commimicate with science 
classrooms. The technology expands 
these classrooms by eliminating the 
limitations of both time and distance 
that would othervnse restrict this type 
of commimication. 

Students involved in a network 
science project about radon were able 
to collect readings in their town, 
compare those readings with other 
towns arovmd the country, and con- 
verse with a scientist at the Environ- 
mental Protection Agency about their 
questions regarding the differences 
they found in wooden and stone 
houses. A text-based study of radon 
would not have provided them with 
the variety of opportimities to consider, 
reconsider, and talk about their ideas 
on this topic. In tlds case, technology 
provided the tools for analysis, as well 
as an opportunity for expanded 
commimication. 

Network science helps a larger 
proportion of students feel 
confident about their ability to 
understand science. 

This point is addressed in several 
ways. Many teachers reported that 
students were motivated and eager to 
participate in the curriculum activities, 
even, or perhaps especially, those 
students who rarely participated in 
science dctss. One teacher told the story 
of a "low ability" student who gained 



eiuirmous credibility in class when he 
proposed his idea about the wide 
discrepancy between the over 120 pets 
owned by his Auburn, Maine, c b'^s and 
the fewer than 25 pels owned by a class 
on his research team in New Orieans. 
Whereas other students had explained 
the difference as something related to 
the weather or the availability of pets in 
Louisiana stores, this student thought 
that perhaps the school was located in 
an area surrounded by government 
housing. He explained that pete are not 
allowed in this type of housing. This 
simple piece of information from a 
fellow student, not from the teacher or 
a textbook, helped students make sense 
of the data and generated a letter to 
find out if this student's theory was 
correct As you might imagine, his idea 
also increased the student's credibility 
with his peers, his teacher, and perhaps 
himself. 

We received other reports of in- 
aeased student self-esteem that are of a 
very different nature. Teachers reported 
that students felt a real sense of owner- 
ship about their work. The most 
extreme examples came firom two 
teachers who reported that their classes, 
both filled with "average" and "below 
average" students, protested upon 
learning that their teacho* was plaiming 
to present the NGS Kids Network 
curriculum at a professional cortference. 
These students argued that because it 
was their work, they should be the ones 
presenting. In one case the teacher took 
the students with him; in the other, the 
students made an acetate filmstrip and 
accompanying audio tape that the 
teacher used in her presentation. 

Science teachers participating in our 
network science programs consistently 
report increased involvement in 
classroom work by all their students, 
particularly those who previously had 
been academic tmderaduevers. One 
teacher in the Star Schools project 
reported that one such student "has 
taken a complete change in personality 
and performance after his [solar] house 
was one of the most successful. He just 
displays a confidence that I didn't see 
before' this unit Learning styles were so 
evident to me while observing my 
students complete the unit It gives me 
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new insight into better teaching 
methods." 

Students' work in classroom 
inquiries is of interest to the 
local community. 

Network science classrooms are 
filled with visitors interested in the 
students' work. The list is long and 
varied and indudes principals, other 
teachers, superintendents, school board 
members, parents, university profes- 
sors and their students, and both 
television and print journalists. These 
visitors are interested in what the 
students are doing, both in terms of the 
telecommvmications activities with 
other schools and the data they are 
collecting, particularly when the data 
are about larger environmental 
concerns such as add rain. This type of 
interest motivates both teachers and 
students, who appredate having ah 
audience for their work. 

The work of a science classroom 
is of interest to the scientific 
community. 

The sdentific community has been 
both supportive of and enthusiastic 
about all our network sdence projects. 
Many sdentists have given hotirs of 
their time exploring the types of 
research students might do that would 
offer valuable data to existing smdies. 
Dr. John Miller, the deputy director of 
the add rain disposition unit of the 
National Oceanic and Atmospheric 
Administration (NOAA) and the unit 
sdentist for the NGS Kids Network 
Add Rain unit, is an excellent example. 
(See "Sdentists in Sdence Education: 
An old idea with a new approach" in 
Hands On!. Spring 1989.) Dr. Miller 
corresponded throughout the unit with 
his elementary-levd "colleagues" cind 
induded the NGS Kids Network data 
as an appendix to the NOAA 1988 
report on add rain. In addition, he 
regularly discussed this project with 
his colleagues around the world. 

Perhaps the most compelling 
example of the serious interest that 
sdentists have in student<ollected data 
took place at the annual meeting of a 
United Nations steering committee 



concerned with the long-range trans- 
mission of air pollutants. As the Uitited 
States representative to this group. Dr. 
Miller presented an overview of the 
various networks in North America 
that are concerned with add rain. 
Within *his context, he introduced his 
colleagues to the NGS Kids Network 
project and was surprised to find that 
they expressed considerable interest in 
the netMTork, wanting to know how 
students made the measurements and 
if the data would be available for 
comparison with NOAA data. We 
have found that this level of profes- 
sional interest is not unusual Sdentists 
are interested in student measure- 
ments, particularly when die data 
cover a wide geographical area. 
In these times filled with reports of 



In these times filled 
with reports of the 
grim realities of 
sdence education, 
our work with 
network sdence 
projects has given 
us considerable 
hope about the 
possibilities for real 
change. 



messing about in sdence. In addition, if 
this type of investigation is linked to 
real enviroiunental concerns, the 
students' work becomes valuable ^ 
commimity information, not just an 
empty school assignment ^ 
One high school student involved in 
a Star Schools project described what 
he felt was the difference between his 
usual sdence class, which was lecture- 
and text-based, and his work in a 
network sdence study. It wasn't like 
writing data charts, because data charts 
are really easy, you just copy things 
from the board or you do math. But 
[in] this [program) you have to sit 
down and think. You had to use your 
brain a little more than usual, so thaf s 
how it was different... I think if we 
hadn't done the radon activity, I 
wouldn't have to use my braLi at all 
the whole year. This is the only time 
that I really had to think.... I had to 
struggle a little, you know, and I'm not 
really used to doing that in sdence." 
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the grim realities of sdence education, 
our work with network sdence projects 
has given us considerable hope about 
the possibilities for real change. 
Instructional materials that center 
around the collection and analysis of 
data not only give students a more 
accurate picture of the sdentific 
enterprise, but they also give students 

an opportimity to examine and ^jjj^ 
construct understanding of the topic 
under study. This approach offers 
them a collective opportunity for 
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What role does assessment play in effective 
science instruction? 



Assessing student progress and instructional quality is an 
important part of science education. One tiung must be made 
dear from the outset assessment encompasses more than 
testing, and much more than standardized testing. It includes 
such techniques as systematic teacher observation and so-called 
"authentic" assessment, in which the tasks assessed more 
closely parallel the learning activities and outcomes iliat are 
desirable in the science dassroom. 

Assessment fulfills many functions, from helping teachers make 
day-to-day classroom dedsions to providing parents, the public, 
and policymakers with information about the overall effective- 
ness of a sdence program. In the classroom, most teachers use 
combinations of teadier-made or end-of-chapter tests and their 
own professional judgments for day-to-day instructional ded- 
sions. They use standardized tests for broader assessment 
purposes. 

New approaches to sdence teaching and learning have 
expanded the role of assessment Consequentiy, some teachers, 
schools, districts, and states are trying out a broader range of 
assessments and using assessment in new ways. For example, 
in an effective sdence classroom, a teacher might use assessment 
to fulfill any or all of the following purposes: 

• Find out what students know and can do before instruction 
begins. 

• Determine how they are progressing toward learning goals. 

• Identify which strategies and thinking processes students x;ise 
to reach answers or condtisions. 

• Pinpoint which questions to ask to determine how well 
students are integrating new information 

• Establish what students have learned after a specific period of 
instruction. 

• Motivate students. 

• Inform parents about their children's learning. 

• Evaluate the effectiveness of special interventions. 



New approaches to 
science teaching and 
learning have expanded 
the role of assessment. 



ERIC 



57 



131 



• Signal to students the areas in which they need to improve. 



Whenttsed 
simplistically or 
inappropriately, 
assessment can. . . 
negatively affect 
students' perceptions of 
themselves as science 
learners. 



• Conununicate teacher expectations about what is important 

• Determine whether they need to alter dieir teaching. 

Schools, school districts, states, and the federal government also 
use assessment to monitor the effectiveness of teachers, schools, 
districts, and special programs, and to make policy decisions. 

Assessment can be a powerful influence on curriculum and 
instruction, for good and for bad. The format of the assessment 
and the uses to which results are put can guide how teachers 
teach and students study, especially when applied to "high 
stakes" decisions such as allocation of resources, admission to 
special programs, or receipt of a high school diploma. When 
xised simpljstically or inappropriately, assessment can drive 
teaching and learning in unhealthy directions and can nega- 
tively aSect students' perceptions of themselves as science 
leameis. 

Many researchers have become deeply concerned that assess- 
ment is not being used well in most science education programs. 
Concerns center around whether assessment instalments, such 
as norm-referenced, standardized tests, are being used for too 
many purposes for which they were not designed, and whether 
the results of tests are being misxmderstood and misapplied. 
Some researchers have asserted that the most common assess- 
ment formats, particularly conventional standardized tests, 
reinforce outmoded or ineffective instructional practices. For 
these and other reasons, many argue that assessment is an area 
of science education that is ripe for reform. MM 



How do standardized tests affect science teaching 
and learning? 



Paper-and-pencU tests, typically standardized multiple-choice 
tests, are the primary assessment tools in most science class- 
rooms. Standardized tests have advantages of efficiency for 
testing lai^ numbers of studsnts, and many people perceive 
them as a more objective and reliable measure of achievement 
than assessments that rely more heavily on individual human 
judgment 

Recently, however, standardized science tests have come xmder 
strong aitidsm Many researchers argue that conventional 
multiple-choice tests are questionable proxies of science knowl- 
edge. Because they do not ask students to generate their own 
answers, the tests do not measure scientific thinking. And 
because they have a single right or wrong answer, they reinforce 
the misleading conception of science as a static body of (ads. 
Further, they do not assess laboratory skills at alL 

Many of the standardized science tests are norm-referenced, 
ranking individual performance against that of a larger group. 
This, too, concerns some analysts, who say it perpetuates the 
notion that only a few students — the top scorers — are smart 
enough to pursue science. Researchers also raise questions 
about how well standardized tests measure the peiformance of 
students who are outside the cultural mainstream 

Of even greater concern to many researchers are the ways in 
which standardized tests may drive instruction in less effective 
directions or affect children in negative ways. Some researchers 
contend that conventional standardized tests, by sampling a 
breadth of content in a superficial and imconnected way, actu- 
ally reward instruction that drills students on low-level facts and 
name recognition. In a high stakes testing situation, some 
studies have found that it is not uncommon for teachers to give 
teacher-made tests that imitate the format of large-scale assess- 
ments. Even schools that have implemented many of the 
refonns described in this document find their efforts imder- 
mined when standardized test scores are what "count." 

Responding to these criticisms, researchers and test publishers 
are developing new fonns of tests, including multiple choice 
tests that require students to justify their answers or that assess 
thinking processes, op)en-ended answer tests, and performance 
assessments. 



Because they do not 
ask students to 
generate their own 
answers, the tests do 
not measure scientific 
thinking. 
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Students might be 
asked to do a 
laboratory experiment 
or to solve a real-life 
problem, 



good or promising methods of 
tsi used in science education? 



Awaxe of the limitations of conventional tests, researchers and 
practitioners are devebping and implementing new forms of 
assessment that seek to better reflect the goals and processes 
recommended for effective science instruction, although these 
new assessments have their own weaknesses. 

Several of these new forms foil imder the vimbrella called perfor- 
mance assessment, in which student create an aitswer or 
product that demonstrates their knowledge or skills. Many of 
these are hands-on in nature. Students might be asked to do a 
laboratory oq)eriment or to solve a real-life problem. Through a 
series of systematized observations and questions, teachers or 
outside evaluators would observe and evaluate both the pro- 
cesses the students use and their tmderstanding of the major 
concepts involved. In this tjrpe of assessment, students may 
work together or separately, using the equipment, materials, 
and procedures th^ would use in good, hands-on science 
instruction. 

Another type of performance assessment that works well for 
students who are verbally or visually oriented is the presenta- 
tion, which could be in the form of an oral presentation or a 
poster paper. This latter format mirrors the way iiumy scientists 
present findings. Authors create posters explaining ideas; these 
posters might include graphs, photos, maps, or drawings, as 
well as some text For example, students might develop a poster 
that explains how a local river evolved to its present state and 
how it nught be protected. 

A related form of assessment is systematic teacher observation, 
in which teachers sdentiiically observe and record student 
behaviors as they carry out meaningful tasks. Still another type 
of performance assessment being pioneered in science requires 
students to amass a portfolio of S\err work over time, which 
might include lab notes, science journals, and other written 
products, or graphs and charts of laboratory findings. 

Research literature is also giving greater emphasis to student 
self-assessment and peer assessment techiuques, in which 
students — for themselves or others — reflect on experience, 
attempt to understand what took place, and make judgments 
about what was learned. 




¥34 



Litegrated computerized systems, which track and report stu- 
dent perfbnnance at the same time students are learning, is a 
growing approach to assessment Paper-and-pendl tests are 
also being revamped to allow for more open-ended and 
student-constructed responses. 

All of tiiese assessments attempt to meet criticisms of current 
testing practices and to focus attention more on processes and 
thinking skills. Many of the performance assessment models 
meet the added criteria, emphasized in research Uterature, that 
assessment should closely resemble instruction and differ only 
in purpose. In addition, tihey provide much richer knowledge 
on which teachers can base instructional decisions. 

Performance assessments have drawbacks, however. First, they 
require staff development for teachers who will implement, 
score, and use the results of the assessments. Secoiid, they are 
considerably more expensive to administer than conventional 
tests. Third, they take more time from the school schedule to 
implement — some may last over several days. Fourth, 
although it is possible to standardize performance assessments 
to allow for comparisons among groups, it is a demanding 
process and is still in the experimental stages for many types of 
assessments. 



Finally, it is difficult to correlate performance assessments from 
one task to another or to use performance assessment to project 
future student performance. And that is one major function of 
standardized testing. 



Many of the performance assessment models meet the added 
criteria, . dhat assessment should closely resemble instruction 
and differ only in purpose. 



10 c 



How can science assessments be better linked to 
instruction? 



Assessment that does a 
good job informing 
instruction should have 
several features. 



Some researchers and educators look forward to the day when 
there will be a "seamless web" of instruction and assessment in 
which assessment is no longer be a distinct activity but is "built 
into" students' regular learning experiences, virtually indistin- 
guishable from instruction. In this vision, students and teachers 
receive ongoing feedback as needed during the instnictional 
process. Assessment for dassroom purposes (as opposed to 
external monitoring purposes) is a routine, non-threatening 
process. 

This vision calls upon students, teachers, and administrators to 
look at assessment in a different way. Teachers must imder- 
stand that the primary information will not be quantitative, but 
will provide a rich portrait of student strengths and weaknesses. 
In addition, assessment to inform instruction does not have to 
compare students to one another, which means that student 
approaches and responses to a problem may look very different 
and still be correct. Students, especially high school students, 
will be encouraged to be the primary users of assessment 
information in their own learning. 

Assessment that does a good job informing instruction should 
have several features. It should measure me processes students 
use as well as the answers they reach. It should measure all of 
the goals of the curriculum, not just a few. It should address 
both group activities and individual ones. It should be devel- 
oped by, or with ample input from, teachers and should include 
teacher professional observation and judgment. Perhaps most 
importantly, it should draw upon information from miiltiple 
assessment sources, including but certainly not limited to tests. 

Finally, assessment to inform teaching and learning should have 
a strong self-evaluation component for both students and 
teachers. 
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What We've Learned About 
Assessing Hands-On Science 



Assessing scientific inquiry is more complex 
than political rhetoric pushing performance 
tasks Indicates, this team of scientists found. 
And, unless carefully crafted and blended into 
science instruction, assessments alone are 
unlikely to boost achievement. 

Richard J. Shavelson and Gail P. Baxter 
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Over the past three years, our 
team of researchers, scientists, 
and science teachers at the 
University of California, Santa 
Barbara, the California Institute of 
Technology, and the Pasadena Unified 
School District has sought to create 
assessments that support good science 
teaching at the elementary level 
(Baxter et al. in press; Shavelson et al. 
1991). In particular, our goal has been 
to develop activities that permit 
students to pursue an experimental 
inquiry focusing on process skills 
(such as observing and inferring) and 
construction of new knowledge (such 
as understanding the effects of insula- 
tors on electric current). In our defini- 
tion, assessments consistent with good 
teaching invite students to perform 
concrete, meaningful tasks such as a 
laboratory experiment to determine, 
for example, how certain kinds of 
insects respond to changes in environ- 
ment. Scoring of performance focuses 
on the reasonableness of the procedure 
used to carry out the investigation, not 
just on the "right answer." 

In our work, we assumed that the 
ideal assessment would be direct 
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observation of a student pursuing a 
scientific inquiry with laboratory 
equipment and materials. This obser- 
vation would be made by scientists 
and science teachers trained to score 
performance in real time. That is, the 
ideal was predicated on the assump- 
tion of the symmetry of teaching and 
testing: an ideal assessment would be 
a good teaching activity and, indeed, 
might even serve as a teaching activity 
when not used for assessment. 

However, observations of individual 
student performance are costly, time 
consuming, and difficult to obtain. 
With the ideal performance assess- 
ment as a benchmark, we developed 
and evaluated alternatives (or surro- 
gates) to the benchmark. They were, in 
order of decreasing verisimilitude, (1) 
lab notebooks in which students 
recorded their procedures and conclu- 
sions; (2) computer simulations of the 
hands-on investigations; (3) short- 
answer papcr-and-pencil problems in 
planning, analyzing, and/or inter- 
preting experiments; and (4) multiple- 
choice items developed from observa- 
tions of students conducting hands-on 
investigations. Finally, we compared 
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the benchmark and surrogate assess- 
ments to a traditional multiple-choice 
science achievement test, the Compre- 
hensive Test of Basic Skills (CTBS). 

We developed and collected data 
using three hands-on investigations: 

• Paper Towels — Given laboratory 
equipment, conduct an experiment to 
determine which of three different 
paper towels soaks up the most water. 

• Electric Mysteries — Given six 
"mystery boxes," determine their 
contents by connecting electric circuits 
to them. 

• Bugs — Determine sow bugs' 
preferences for various environments 
(for example, dark or light, dry or wet). 

The performance of more than 300 
students, some experienced in hands- 
on science and some who had received 
minimal science instruction from a 
textbook on health, was observed and 
scored in real time by science educa- 
tors. In addition, all students 
completed corresponding notebooks, 
computer simulations, paper-and- 
pencil measures, and the CTBS. 

Four questions guided our research: 
(1) Could reliable measures of hands- 
on performance and of surrogate 
assessments be developed? We wanted 
these measures to permit a wide 
variety of student responses found 
when doing science. We also wanted 
to develop a method to score perfor- 
mance that captured the diversity of 
procedures and put them on a common 
scale. (2) Could the performance of 
students with different instructional 
experiences (hands-on vs. textbook) 
be distinguished? We expected 
students experienced with hands-on 
science to perform better on the 
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benchmark and closest alternatives 
than students in a textbook cuiriculum 
program. (3) Do the performance 
assessments provide information about 
student achievement not available 
from traditional multiple-choice 
science tests? If not, periiaps nothing 
new had been developed. (4) Do the 
suiTOgate assessments capture the infor- 
mation gained from the benchmadc? If 
so, then dollars and time can be saved 
in administration and scoring. 

Hands-On Investigations 

Students conducted the three investi- 
gations in approximately 1 1/2 hours 
while being observed by a scientist or 
science educator trained to score 
student performance. 

Paper Towels. Snidents used a 
laboratory setup to determine which of 
three paper towels held the most and 
least water. Students were told that 
they could use all or some of the 
equipment, whatever they needed. A 
scheme was developed to score the 
diversity of procedures used to cany 
out the investigation on a common 
scale (see fig. 1). An outstanding 
investigation completely saturated 
each towel, determined the amount of 
water each held by a method that was 
consistent with the way the towel was 
wened. and the entire procedure was 
done carefully. For example, a snident 
might saturate the towel in the pitcher 
of water and weigh it in the scale, 
carefully removing the e,xcess water in 
the scale after weighing each towel. 
Carelessness, inconsistencies in the 
method of wetting the towel and 
measuring the results, incomplete 
saniration, and irrelevant methods 
resulted in less than outstanding 
scores. The scoring scheme identified 
the procedure used and could thereby 
characterize performance in terms of 
both processes and outcomes. More- 
over, several different performances 
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High-quatiiy assessments throughout a course are vital if teachers are to aceurattly 
appraise performance. 
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PAPER TOWELS INVESTIGATION— HANDS-ON SCORE FORM 



Students 



. Observer. 



. Score . 



. Script. 



1. Method A. Container B. Drops 

Pour water irt/put towel in 
Put towel irt/pour water in 
1 pitcher or 3 beakers/glasses 

2. Saturation A. Yes B. No C. Controlled 

3. Determine Result 

A. Weigh towel 

B. Squeeze towel/measure water (weight or volume) 

C. Measure water in/out 

D. Time to soak up water 

E. No measurement 

F. Count » drops until saturated 

G. See how far drops spread out 

H. Other 

4. Care in Measuring Yes No 



C. Tray (surface) 
towel on tray/pour water on 
pour water on tray/wipe up 



S. Correct Result 


Yes 


No 








Grede 


Method 


Seturate 


Determine 


Care In 


Correct 








Result 


Meeturtng 


Answer 


A 


Yes 


Yes 


Yes 


Yes 


Yes 


B 


Yes 


Yes 


Yes 


No 


Yes/No 


C 


Yes 


Yes/Controlled 


Error 




Yes/No 


0 


Yes 


No 


Missing 




Yes/No 


F 






■ No Attempt - 
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could result in the same letter grade. 

Bugs. Students used laboratory 
apparatus to determine the preferences 
of sow bugs for environments that 
were light or darlc, damp or dry, or 
some combination of the four. The 
scoring scheme resembled the one 
used in the towels investigation. 

Electric Mysteries. This investiga- 
tion was a bit different from the 
others. Students were given batteries, 
bulbs, and wires and asked to deter- 
mine the circuit components hidden in 
a set of mystery boxes (see fig. 2). 
Performance was scored on the basis 
of (1) their determination of the 
contents of each box and (2) the 
sequence of tests they conducted to 
determine the contents. 

Notebook Surrogates 

For each investigation smdents were 
asked to keep notebooks describing 
their investigation so that a friend 
could repeat it exactly. Notebooks 



have several advantages for large- 
scale assessment Through the use of 
notebooks instead of expert observers, 
many students can be tested with 
hands-on investigations. Notebooks 
provide an opportunity for students to 
express themselves in writing, an 
important skill in science and a way of 
integrating curricular areas. And the 
notebooks can be scored quickly — in 
about one to two minutes per smdent 
Notebooks, then, preserve much of the 
hands-on investigation while reducing 
time and cost of expert observers. 
They also capture the rather inventive 
namre of the investigations and ways 
of reporting on them. 

Computer Simulation Surrogates 

We developed our own computer 
simulation for the Electric Mysteries 
and Bugs investigations to replicate, as 
nearly as possible, the hands-on inves- 
tigations. (The Paper Towels investi- 
gation could not be simulated 
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HANDS-ON ELECTRIC MYSTERIES INVESTIGATION 



Find out what is in the six mystery boxes A, 
8, C, D, E. and F. They have five different 
things inside, shown below. Two of the 
boxes will have the same thing. All of the 
others will have something different inside. 



When you find out what is in a box, fill in th« 
spaces on the following pages. 



Box A: Has . 



inside. 



Two batteries: 



A wire: 



Draw a picture of the circuit that told you what 
was inside Box A. 



A bulb: 



A banery and a bulb: 



Nothing at all: 




How could you tell from your circuit what was 
inside Box A? 



Do the same for Boxes B, C, D, E, and F. 



You can use your bulbs, batteries, and wires 
any way you like. Connect them in a circuit 
to help you figure out what is i.'<side. 
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adequately.) For the electric circuits 
investigation, students used a Macin- 
tosh computer with a mouse to 
connect circuits to the mystery boxes 
to determine their contents. The inten- 
sity of the luminosity of the bulb in a 
real external circuit was accurately 
simulated. Students could coimect a 
multitude of circuits if they so desired. 
Alternatively, they could leave a 
completed circuit on the screen for 
comparison. A teacher-directed tuto- 
rial prior to the test provided smdents 
with instructions on how to record 
their answers, erase wires, save their 
work, or look at a previous page of 
their work on the screen. The 
computer recorded every move the 
smdent made. The bug simulation was 
constructed similarly, though it was 
not possible to record every move the 
smdent (and the bugs) made. 

Computer simulations have a 
number of desirable properties for 
assessment They are less costly and 
time consuming to administer than 
hands-on assessments, though develop- 
ment costs are considerable. Smdents 
can be tested in groups by a parent or 
volunteer who has been briefed on how 
the simulations work. Smdent perfor- 
mance can be scored quickly and easily 
using the scoring system developed for 
the hands-on investigations. In addi- 
tion, a computer simulation maintains a 
full record of performance, so that 
teachers and/or smdents can review 
problem-solving processes. Fmally, 
smdents experiment with the tech- 
nology, discovering solutions to prob- 
lems that they might not have found 
with other types of assessments. In 
other words, the computer assessment 
provides a good instructional tool. 

Pencil-and-Paper Surrogates 

Short-answer and multiple-choice 
questions were chosen to parallel, in 
content the three hands-on investiga- 
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tions. For the Electric Mysteries short- 
answer questions, students received a 
pictorial representation of a problem 
similar to one encountered during the 
hands-on investigation. For example, 
students might be asked how they 
would determine the contents of a 
particular mystery box without 
looking inside. For the Paper Towels 
and Bugs questions, students were 
given descriptions of portions of the 
investigations and questioned about 
the control of variables, the setup of 
the experiment, or the best method to 
use for measuring results. 

Multiple-choice questions began 
much like the short-answer questions. 
Rather than formulate a response, 
students chose among four alterna- 
tives, all of which were based on 
observed performance. For example, 
an Electric Mysteries question 
presented alternative circuits 
connected to a box and asked students 
to indicate which circuit would tell 
them what was inside the box. 

The paper-and-pencil surrogate 
assessments differ fundamentally from 
the other surrogates; they do not 
respond to the actions taken by the 
students. Even if a paper-and-pencil 
test provided immediate wntten feed- 
back on a decision made by a student 
(Shulman and Elstein 1975). we doubt 
it would have the same impact as the 
feedback from the real-life (hands-on) 
or lifelike (computer) assessments. We 
may not be able to develop paper-and- 
pencil surrogates that overcome this 
limitation. 

Findings 

We found the process of creating 
performance assessments symmetric 
with good teaching activities to be 
time consuming, requiring consider- 
able scientific and technological 
know-how. Development of quality 
performance assessments requires 
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If teachers teach 
to poorly constructed 
assessments, these 
assessments are likely 
to distort good hands- 
on science teaching. 



multiple iterations through a sequence 
of development, tryouts with students 
(getting their thoughts and comments), 
and revision. Short-circuiting this 
process leads to ill-conceived and 
poorly constructed assessments. Such 
assessments are as likely to lead to 
poor teaching — if teachers teach to 
the test, and they do (Smith 1991) — 
as are ill-conceived and poorly 
executed classroom activities. 

Once the performance assessments 
were constructed, our research posed 
four questions about them: (1) Can 
they provide reliable measurements? 
(2) Are they sensitive to students* 
instructional experiences? (3) Do they 
provide achievement information that 
differs from u^ditional measures? (4) 
Are they interchangeable? Our find- 
ings (some good news, some bad 
news) balance the political rhetoric 
pushing implementation of perfor- 
mance assessments with a cold reality. 

Reliability. Raters can reliably eval- 
uate students' hands-on performance 
on complex tasks in real time. Reliabil- 
ities are high enough (above 0.80) that 
a single rater can provide a reliable 
score. But task-sampling variability is 
considerable. Some students perform 
well on one investigation while others 
perform well on a different investiga- 
tion: general "expertise" is more in the 
mind of the beholder than in perfor- 
mance. To get an accurate picture of 
individual student science achievement, 
the student must perform a substantial 
number of investigations — perhaps 
between 10 and 20. 

1^10 



Instructional history. Performance 
assessments can distinguish students 
with different instructional histories. 
Assessments that are closely linked to 
a specific domain of knowledge (for 
example, electric circuits/electricity) 
are more sensitive to performance 
differences than more general process 
assessments (for example, the Paper 
Towel investigation). But to be sensi- 
tive to instructional history, perfor- 
mance assessments must be carefully 
crafted to measure more than science 
process. They need to measure the 
application of both concepts and 
procedures. 

Relation to multiple-choice tests. 
The good news is that performance 
assessments do not duplicate informa^^ 
tion about student achievement 
provided by u^ditional tests (averag^^' 
correlation is about 0.45). They tap 
somewhat different aspects of achieve- 
ment. But we are not sure what 
aspects of achievement multiple- 
choice tests or performance assess- 
ments do and do not tap. Indeed, a 
combination of indicators (multiple- 
choice, performance assessments, and 
others) may be needed to provide a 
complete picture of achievement. 

Interchangeability of surrogates. 
Certain surrogate assessments appear 
to be interchangeable with their corre- 
sponding benchmark. This is espe- 
cially true of notebooks for student- 
level assessment. Computer 
simulations are interchangeable with 
their corresponding benchmarks if the 
intent is to estimate classroom-level 
mean performance. But for individual 
students, measures of science achieve- 
ment are highly sensitive not only to 
the investigation used (for example. 
Bugs vs. Electric Mysteries), but als^ 
to the method used to measure perfor- 
mance (for example, observation vs. 
simulation). Some students' scores 
depend on the particular investigation 
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(Electric Mysteries, Paper Towels, 
Bugs) and on the partictilar method 
used to assess performance (observa- 
don, notebook, computer simulation, 
paper-and-pencil). Indeed, each 
combination of investigation and 
method provides different insight into 
what students know and can do. 

Conclusions. A fundamental 
assumption made by policymakers 
(for example. Bush 1991) and other 
education reformers (for example, 
Wiggins 1989) is that, by changing the 
namre of the achievement test, 
teachers who teach to the test will 
have to change and improve their 
teaching. Our experience with perfor- 
mance assessments suggests that this 
assumption is, at best, half true. 
Teachers will indeed change the way 
they teach if held accountable by 
performance assessments. But, 
without high quality assessments and 
staff development in quality instruc- 
tion, they very weU may not improve 
their teaching. Moreover, one-shot, 
end-of-year tests cannot provide 
adequate information on individual- 
level smdent achievement. Continuous 
assessment throughout the course of 
instruction is needed to accurately 
reflect smdent science achievement. 

Assessment Development 

Performance assessments are very 
delicate instruments. They need to be 
carefully crafted, each requiring a 
specially developed or adapted scoring 
method. Shortcuts taken in devel- 
oping these assessments will likely 
produce poor measuring devices. If 
these insmiments are used to judge the 
quality of education in classrooms — 
and they v/ill be used for that purpose 
— then teachers will teach to the test. 
If teachers teach to poorly constructed 
assessments, these assessments are 
likely to distort good hands-on science 
teaching. 
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This conclusion was brought home 
to us in our observations of someone 
we consider to be an outstanding 
hands-on science teacher. In her class, 
a unit on electricity is taught in a 
series of lessons. Small groups of 
smdents conduct hands-on investiga- 
tions, not unlike the ones we have 
developed. Smdents keep notebooks 
and draw conclusions based on the 
outcomes of their experiments and 
their discussions with other groups of 
"scientists" in the class. 

Excited about hands-on science 
teaching, our teacher volunteered her 
class for pilot testing California's new 
hands-on science assessments. These 
assessments were constructed under 
severe time and cost constraints, and 
consequentiy involved minimal trials 
with smdents. To meet testing time 
and space constraints, and in recogni- 
tion of differing curriculums in the 
state's elementary schools, the assess- 
ments were accompanied by detailed 
directions to smdents. Students read 
the instructions, followed explicit 
procedures, and reported what they 
found in the spaces provided. The 
assessments were more like recipes 
(first do this step, then do this step) 
than like scientific investigations. 

Based on her experience with these 
assessments, our exemplary teacher 
began to modify her teaching to corre- 
spond to the state's pilot assessments. 
Rather than open-ended, group 
problem solving, emphasis was placed 
on reading instructions and carefully 
following format, carrying out a set of 
required procedures, and recording 
fmdings in a prespecified format in 
notebooks. (For example, students 
were admonished to "be sure to write 
complete sentences or the state will 
grade you down.") The essence of 
doing science was becoming one of 
following procedures. The story ends 
happily. After we pointed out what 
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was happening, the teacher went back 
to her "old ways." 

Staff Development 

Unless provided the scientific and 
pedagogical knowledge required for 
hands-on science teaching, teachers 
may very weU flounder in their 
attempts to match their teaching to the 
testing. Once again, an anecdote may 
bring home this conclusion. One of the 
teachers whose class was participating 
in our research knew of the Paper 
Towels investigation. Teaching to the 
test, she instructed her students to 
samrate the towels completely, using 
the timer to ensure that each towel was 
saturated for at least 10 minutes — a 
total of 30 minutes for saturation! In 
reality, the towels could be samrated 
in a matter of seconds. This led her 
smdents, when tested, to perform in a 
clearly stylized manner. She had 
informed the students, perhaps unin- 
tentionally, that science is a set of 
precise steps that must be carried out 
invariably, regardless of whether they 
make sense. Her approach was not 
particularly conducive to scientific 
exploration. Although the teacher 
could teach the smdents how to "do" 
the experiment, what was missing was 
an understanding of the essence of 
doing science. 

Curriculum-Embedded Assessments 

To obtain sufficiently large samples of 
smdent performance, assessments may 
need to be taken throughout the 
academic year. For example, students 
might receive the Electric Mysteries 
assessment embedded in the activities 
composing a unit on electricity. Like- 
wise, assessments would be embedded 
in three or four other units as well. 

The embedded assessments have 
several desirable characteristics. They 
provide almost immediate feedback to 
teachers on their smdents' perfor- 
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mance. and on how this perfonnance 
compares with that of students in 
comparison schools. Moreover, the 
assessments reinforce hands-on 
teaching and learning. Nowhere is the 
symmetry between teaching and 
assessment more apparent than with 
embedded assessments. 

Embedded assessments do not 
preclude an end-of-year examination. 
The latter provides both additional 
information on achievement and an 
external audit ensuring data credibility 
to the various audiences interested in 
educational accountability. 

Will Achievement Improve? 

Results of our research suggest that 
the political rhetoric calling for imme- 
diate reform of national, state, and 
local testing systems far exceeds 
current technological capability and 
ignores educational and social conse- 
quences. No doubt assessment systems 
will be changed in the very near 
future. The politicians will have their 
day. We suspect that the initial impact 
will be to change classroom activities 
and the nature of assessment, possibly 
embedding assessments in classroom 
activities. However, without quality 
instrumentation and extensive staff 
development, the bottom line — 
achievement — will probably not 
change. 

The nation may be capable of 
producing the kind of assessment 
systems currently envisioned by the 
rhetoric if the politicians stick to their 
guns and do not retreat to the usual 
multiple-choice testing. Politicians 
need to provide resources for 
preparing beginning and current 
teachers for teaching and testing 
reforms, and for fine-tuning the 
assessments through research, social 
debate, and revision. With the 
symmetry between assessment and 
Q teaching firmly established, the 
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bottom line — achievement — may 
very well improve. □ 

Authors' note: This research was 
supported by a grant ficom the National 
Science Foundation (No. SPA-8751511). 
The ideas presented reflea those of the 
autbon and not necessarily the NSF. 
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NCTM's Standards: A Rallying Flai 
For Mathematics Teachers 



Thomas A. Romberg 

The development of 
the Standards is the 
story of a professional 
community recognizing 
a need, reaching 
consensus through 
review and analysis, 
and conceiving a clear 
vision and a strategy 
for mathematics reform. 
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Curriculum and Evaluation Stan- 
.dardsfor School Mathematics 
has been hailed is the exemplar 
of what is needed in all curric- 
ular areas if we are to reform 
American education during the 
coming decade (National Council of 
Teachers of Mathematics 1989). As 
chair of the commission that produced 
this 258-page document, my intent 
here is to provide an account of how 
the mathematical sciences education 
community developed the Standards. 

Origin of the Standards 

The strategy we used came about as a 
result of two conferences held in 1983 
in response to the perceived crisis in 
education, as voiced in A Nation at 
Risk and Educating Americans for the 
Twenty-First Century.^ The first 
conference was sponsored by the 
Conference Board of the Mathemat- 
ical Sciences and funded by the 
National Science Foundation; the 
other was joindy sponsored by the 
National Council of Teachers of Math- 
ematics (NCTM) and the University 
of Wisconsin and funded by the U.S. 
Depanment of Education. The dozen 
or so recommendations contained in 
the reports of both meetings were 
strikingly similar.- Each suggested the 
development of a "new content frame- 
work" for the K-14 mathematics 
curriculum. 

In 1986 the National Council of 
Teachers of Mathematics organized 
the Commission on Standards for 
School Mathematics to prepare a draft 
document. Composed of classroom 
teachers, supervisors, mathematics 
educators, and mathematicians, the 
Commission formed four working 
groups, one each to address 
curriculum for grades K-4, 5-8, 9-12. 
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and a group to address evaluation. In 
developing the draft, the groups 
consulted reports and background 
papers related to the calls for reform; a 
variety of research reports;' recent 
state curriculum documents (Cali> 
fomia, Oregon, Wisconsin); 
curriculum documents from other 
countries (Australia, The Netheriands, 
Japan, The United Kingdom); and two 
summary papers outlining the perspec- 
tives and the tasks to which we were 
committed. 

During the summer of 1987, the 
groups prepared and distributed 
10,000 copies of our draft for review 
by NCTM and the Mathematical 
Sciences Education Board. We 
received more than 2,000 suggestions 
from parents, business leaders, 
teachers, and mathematicians. We also 
asked other professional mathematics 
organizations to critically review and 
endorse the document. In 1988, the 
wridng groups met again to revise the 
document, which, after additional 
review and editing, was published in 
1989. Also in 1988, NCTM created 
several more working groups to 
develop the Professional Teaching 
Standards, which were published in 
1991. Three facts are important in this 
sequence of events: 

1 . The Standards were created as a 
consequence of scholarly review and 
analysis. 

2. Review and consensus involving 
many groups were paramount in the 
process. 

3. Using its own resources, a profes- 
sional organization produced the 
document. No authority to develop 
standards was demanded by a govern- 
mental agency or corporate group. The 
professional community decided it 
was needed and took the initiative. 



The Reform Vision 

The vision of school mathematics 
expressed in the Standards is captured 
in this statement: "All students need to 
learn more, and often different, mathe- 
matics. . . . Instruction in mathematics 
must be significantly revised" (NCTM 
1989, p. 1). This statement has five 
implications: 

1. Teaching mathematics to "all 
students" emphasizes that anyone who 
is to be a productive citizen in the 21st 
century must be mathematically 
literate — including not only "talented 
white males" but all underrepresented 
groups. 

2. "More mathematics" implies that 
all students need to learn more than 
how to manipulate arithmetic routines. 
At present, nearly half of American 
students never study any mathematics 
beyond arithmetic. 

3. "Often different mathematics" 
indicates that all students need to learn 
concepts from algebra, geometry, 
trigonometry, statistics, probability, 
discrete mathematics, and even 
calculus. 

4. 'To learn" means more than to 
memorize and repeat. Learning 
involves investigating, formulating. 



representing, reasoning, and using 
appropriate strategies to solve prob- 
lems, and then reflecting on how 
mathematics is being used. 

5. "Revised" instruction implies that 
teachers and students need to envision 
mathematics classrooms as discourse 
communities where conjectu'cs are 
made, arguments presented. Miategies 
discussed, and so forth. 

The content that all students should 
have an opportunity to learn is 
described in detail in Curriculum and 
Evaluation Standards for School 
Mathematics. To capture the impor- 
tance of both technical and reflective 
knowledge, we chose the term mathe- 
matical power, which envisions a citi- 
zenry : : d society empowered by 
mathematics. For the individual, math- 
ematical power means having the 
experience and understanding to 
participate constructively in society. 
For a culture to be mathematically 
powertul. its citizens must have the 
understanding and experience to 
undertake the routine tasks of 
everyday life, to operate as a demo- 
cratic society, and to progress as a 
civilization. This means the acquisi- 
tion, in society, of a critical mass of 
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understanding, experience, and atti- 
tude, in addition to a substantia] range 
of special expertise. 

Thus, students need to learn to v; 
mathematics, to reason and ;ommu'ni 
cate mathematically, and to become 
confident in their power to use mathe- 
matics coherently to make sense of 
problematic situations in the worid 
around them. Hence, the Standards 
advocate the application of four basic 
standards to all of the content stan- 
dards proposed: mathematics as 
problem solving, mathematics as 
reasoning, mathematics as communi- 
cation, and mathematical connections 
(making linkages within mathematics 
and between mathematics and the real 
wodd). 

The Reform Strategy 

The Commission's intent to develop a 
document titled Standards was delib- 
erate. According to the Oxford Amer- 
ican Dictionary, a standard is "a thing 
or quality or specification by which 
something can be tested or measured." 
"the required level of quality." or "^^j^ 
distinctive flag" (1980, p. 666). Foru^^ 
a standard specified criteria to judge 
the quality of the mathematics 
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curriculum and methods of evaluating 
it. In order to judge quality, we 
decided that there must be agrsed- 
upon expectations. Thus, standards 
become statements about what is 
valued. 

Historically, groups have adopted 
standards to (1) assure quality. (2) 
indicate goals, and (3) promote 
change. The NCTM Commission on 
Standards was created to achieve each 
of these three objectives. In the United 
States, any group or individual can 
produce and sell a mathematics text- 
book, test, or curriculum guide. 
Although most such efforts are well 
intentioned, the recent proliferation of 
manipulatives, software, and modules 
have made minimal curricular stan- 
dards a necessity. The Commission 
focused on the development of stan- 
dards as statements of expectations or 
as "criteria for excellence." However, 
part of our broader strategy was to 
provide mathematics teachers "a 
distinctive flag" to justify their 
demands for change. 

The need for such a document was 
made explicit at a 1983 conference of 
publishers, held at the University of 
Wisconsin. Since we live in a supply- 
and-demand economy, the publishers 
argued, if the mathematics community 
wanted different texts and tests, a 
demand would have to be created. To 
respond to this challenge, the mathe- 
matical sciences education community 
has followed a seven-step iterative 
strategy to reform school mathematics. 
The steps and the relationships 
between them are as follows: 

1 . Before any plan can proceed, a 
need for change must be established. 
First, evidence must be presented that 
the current system is not effective. 
Second, planners must decide whether 
the problem is the result of a lapse in 
quality control, a design flaw, or a 
combination of both (Cuban 1988). 
Third, if a new design is needed, plan- 
ners must be aware of the traditions 
that have to be challenged. The need 
for a new blueprint for mathematics 
education has been well documented.'* 

2. Vision is a key factor if a new 



In economics, a 
product is judged 
of good quality 
if it satisfies 
customer needs 
while making the 
company a profit. 

design is needed. Design change 
involves not only eliminating or 
replacing inadequate materials or prac- 
tices, but creating a new system. Real- 
izing a vision necessarily includes 
consideration of values, goals, and 
standards. The Standards were 
designed to fulfill this vision. 

3. Planning includes involving 
everybody in a system or school in 
arriving at consensus about the details 
of long- and short-range plans (with 
timetables) for change. It is at this step 
that demand is created. 

4. The next step involves identi- 
fying specific elements of the system 
to be targeted for change (curriculum 
materials, instructional methods, 
examinations, teachers, technology) 
and prioritizing them. 

5. Any system depends on 
suppliers. In a supply-and-demand 
society, schools must demand that 
suppliers (textbook publishers, testing 
companies, staff developers, univer- 
sity teacher education programs) 
contribute the ingredients necessary 
for the desired changes in elements. 

6. Then it is time to operationalize 
the new materials, procedures, and 
programs. Feedback from this trial 
phase is matched with the vision and 
the plan for judgment, and revisions 
may be made. 

7. Finally, a product (a curriculum, 
an instructional procedure, assessment 
materials) is developed. In economics, 
a product is judged of good quality if 
it satisfies customer needs while 
making the company a profit. In 
education, quality should be judged in 
terms of what students are able to do 



and whether this meets society's 
needs. Again, via feedback, we now 
return to the vision and our goals and 
objectives to update or revise the 
vision, plans, specific elements, and so 
forth. 

Our intent is that school staffs at all 
levels use this seven-step strategy to 
develop a reform curriculum for K-12 
mathematics. In panicular, each math- 
ematics teacher must understand the 
need for change, own the vision, 
participate in planning, become a 
spokesperson for the demands for nev» 
products and processes, try out new 
materials, and judge student progress 
toward the reform vision. Teachers 
need to understand the critical role 
they play in curriculum reform efforts. 

Status of the Standards 

Implementation of the Standards 
means a number of things need to 
occur using the documents to plan 
change, making demands of suppliers, 
enhancing the professional status of 
mathematics teachers and educators, 
and empowering teachers to voice 
their views. Planning for change in 
education takes place on at least three 
organizational levels. 

Reform initiatives at the national 
level are being influenced by the Stan- 
dards. They were a focal point in the 
Bush administration's strategy for 
school reform and were adopted by 
the National Education Goals Panel. 
At the state level, reform plans should 
go beyond goal statements and general 
expectations to curricular frameworks 
for schools. Several states (California 
and Texas, for example) have recently 
produced frameworks based on the 
Standards, and several others are in 
the process of developing them. In 
addition, many states (19, at last 
count) have initiated new assessment 
practices, in part to align testing with 
the goals of the reform effort. 

At the local district level, plans for 
change should go beyond specifying 
materials and resources to designating 
how teachers and snidents should 
interact with them. It is encouraging 
that many districts have formed 
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curriculum committees; however, it is 
hard to estimate how many of these 
districts have developed implementa- 
tion plans. 

The evidence that the demands for 
change are being heard is clear. The 
Nadonai Science Foundadon is 
funding a variety of curriculum and 
teacher enhancement projects. The 
Department of Educadon is encour- 
aging the use of Eisenhower funds to 
assist teachers in developing new 
materials. California, Texas, and other 
states are urging publishers to develop 
new texts to meet their frameworks. In 
fact, at current professional meetings, 
book publishers are claiming either 
that their current materials meet the 
criteria set forth in the Standards or 
that they are developing new materials 
to meet them. Test publishers, too, are 
busy developing new instruments to 
be aligned with the Standards. 

The professional status of mathe- 
maucs teachers and educators and 
their empowerment has also been 
enhanced. The creation of the Mathe- 
matical Sciences Education Board 
(another of the recommendations) 
has provided the mathematics commu- 
nity with a national agency to repre- 
sent its constituents. Membership in 
national and local mathematics 
teachers' organizations is at an all- 
time high, and attendance at meetings 
has set records every year. Finally, an 
increasing number of mathematics 
teachers are being asked to testify at 
hearings and serve on national and 
state committees dealing with mathe- 
matics education. 

The impact of the Standards docu- 
ment since its publication three years 
ago has gone far beyond our expecta- 
tions. However, the hard pan of 
systemic change is yet to happen. 
Only when the plans and new products 
are being used in classrooms by 
teachers who own the philosophy and 
vision expressed in the Standards will 
real change begin to occur. 

Challenges That Remain 

To carry out the vision of a reformed 
school mathematics curriculum, poll- 



In education, 
quality should be 
judged in terms of 
what students are 
able to do and 
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cymakers and school staffs face a 
number of issues and problems. 

The meaning of "standards"? To 
many persons, the term standards 
implies measuring student perfor- 
mance. Although not a serious issue, 
the use of the word to indicate a 
vision that mathematics teachers can 
rally around is unusual and requires 
explanation. 

Nominal change. To appease 
demands for change, producers often 
change labels but not substance. For 
example, since 1990 the authors of the 
National Assessment for Educational 
Progress have claimed substantial 
changes in items in the test battery in 
light of the reform expectations. In 
fact, both the 1990 and 1992 batteries 
show little alignment with the NCTM 
Standards (Romberg et al. 1992). 

What is mathematics? "Most of the 
population perceive mathematics as a 
fixed body of knowledge long set into 
final form. Its subject matter is the 
manipulation of numbers and the 
proving of geometrical deductions. It 
is a cold and austere discipline which 
provides no scope for judgment or 
creativity" (Barbeau, 1989, p. 2). The 
aims of teaching mathematics need to 
go beyond this narrow view to 
empower learners to create their own 
mathematical knowledge; to reshape 
mathematics, at least in school, to give 
all groups more access to its concepts 
and to the power its knowledge brings; 
and to bring the social contexts of the 
uses and practices of mathematics into 
the classroom. When mathematics is 
seen in this way, it can then be studied 
in living contexts tiiat are meaningful 
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to learners, including their languages, 
cultures, and everyday lives, as well as 
their school-based experiences. 

Tlte "saber-tooth tiger" content. 
Topics should be included because of 
their inherent worth, not because they 
have "always been pan of the 
curriculum." Peddi well's (1939) satir- 
ical tale of continuing to teach 
students techniques to scare saber- 
tooth tigers with fire long after the 
tigers have become extinct is analo- 
gous to the continued expectation that 
students master interpolation of loga- 
rithms, square roots, long division, 
and myriad other routine procedures 
long after computers have automated 
them. Further, room needs to be made 
in the curriculum for new, or newly 
important, topics such as statistics or 
discrete mathematics. 

Teacher independence and isola- 
tion. One tradition of schooling is that 
teaching happens behind closed doors 
(Metz 1978). Such independence 
allows teachers to take risks and be 
creative. Taken to an exticme, inde- 
pendence often leads to isolation. 
Porter ( 1988) refers to one conse- 
quence of independence as a 
curriculum "out of balance." Elemen- 
tary school mathematics is routinely 
taught in such a way that students are 
repeatedly exposed to the same 
content from one grade to the next, 
with skills typically receiving 10 times 
the emphasis given to either concep- 
tual understanding or application. The 
reform vision sees as the norm a 
balanced curriculum arrived at via 
teacher collaboration, joint planning of 
lessons, and shared judgments about 
student performance. 

The "hidden " curriculum. Probably 
the biggest challenge of reform for 
parents, administrators, and even 
teachers is its threat to school routines 
that are based upon existing architec- 
ture, organization, and management. 
Reform ideas are challenging such 
notions as: 

■ "Math is what we do to quiet kids 
down after recess." 

■ "Drill on procedures teaches 
students how to follow rules." 

February 1993 
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■ "Success on a math test is essen- 
tial for tracking." 

■ "Math should not be fun. It 
teaches students about drudgery and 
failure." 

■ "Cunicular changes must improve 
standardized test scores." 

Isn 't this a repeat of the "new 
math "? This is a natural question, 
given the failure of post-Spumik 
aaempts to develop a new mathe- 
matics curriculum. The answer is: 
NO! Hie "new math" was an elitist 
attempt by university mathematicians 
to better prepare college-bound mathe- 
maucs students for a changed colle- 
giate curriculum (Romberg 1990). The 
current movement focuses on mathe- 
matics for all students and is being 
organized by the teachers of mathe- 
matics at all levels. 

How is this effort related to other 
reforms? Although the objectives of 
the mathematical sciences education 
community were developed indepen- 



dently of many other programs, if they 
embrace the same assumptions about 
subject maaer content and pedagogy 
as other change plans, then they 
should be compatible. Systemic 
change based on the same ideas is 
needed. 

A Vision and a Strategy 

There is no question that the current 
school mathematics curriculum is out- 
of-date. The majority of American 
students do not have an opportunity to 
become empowered mathematically 
even though our culture demands 
mathematical literacy of its citizens. 
Creating new curricuiums will take 
time, hard work, commitment, 
patience, and persistence. 

In Curriculum and Evaluation Stan- 
dards for School Mathematics, the 
mathematical sciences education 
community has proposed both a vision 
of a mathematics curriculum designed 
to meet the need and a strategy that 



districts and schools can follow to 
construct a curriculum consistent with 
this vision. For teachers, the Standards 
provide a flag to rally around as they 
fulfill their essential role in mathe- 
matics refonn. ■ 

'National Commission on Excellence in 
Education. (1983). A Nation At Risk: The 
Imperative for Educational Reform. (Wash- 
ington. D.C.: U.S. Government Printing 
Office); and National Science Board 
Commission on PrecoUege Education in 
Mathematics, Science, and Technology, 
( 1983), Educating Americans for the 
Twenry-First Century: A Plan of Action for 
Improving the Mathematics, Science, and 
Technology Education for All American 
Elementary and Secondary Students So 
That Their Achievement Is the Best in the 
World by 1995, (Washington, D.C.: U.S. 
Government Printing Office). 

•The two reports are: New Goals for 
Mathematical Sciences Education, report of 
a 1983 conference sponsored by the 
Conference Board of the Mathematical 
Sciences, Airlie House, Warrenton, Va., 
(Washington, D.C,: CBMS); andT.A. 
Romberg, (1984), School Mathematics: 



Options for the 1990s. Chairman's Report 
of a Conference, (Washington. D.C: U.S. 
Government Printing Office). 

'Summarized in a three-volume report: 
T.A. Romberg and D.M. Stewart. (1987), 
Monitoring of School Mathematics: Back- 
ground Papers (Madison: Wisconsin 
Center for Education Research). 

*In particular, see Everybody Counts, 
(1989), (Washington, D.C.: Mathematical 
Sciences Education Board). 



References 

Barbeau. E. J. (1989). Mathematics for the 
Public. Paper presented at the meeting of 
the International Commission on Mathe- 
matical Instruction, Leeds University, 
Leeds, England. 

Cuban, L. (1988). "A Fundamental Puzzle 
of School Reform." Phi Delta Kappan 
69,5:341-344. 

Metz, M. H. (1978). Classrooms and Corri- 
dors. Berkeley: University of California 
Press. 

National Council of Teachers of Mathe- 
matics. (1989). Curriculum and Evalua- 
tion Standards for School Mathematics. 
Reston, Va.: NCTM. 



Oxford American Dictionary. (1980). New- 
York: Oxford University Press. 

Peddiwell, J. A. (1939). The Saber-Tooth 
Curriculum. New York: McGraw-Hill. 

Porter. A. (1988). A Curriculum Out of 
Balance: The Case of Elementary School 
Mathematics (Research Scries No. 191). 
East Lansing, Mich.: Michigan State 
University, Institute for Research on 
Teaching. 

Romberg, T. A. (1990). "New Math" Was a 
Failure— Or Was It? UME Trends 2, 6: 
1-3. 

Romberg, T. A., M. Smith, S. Smith, and L. 
Wilson. ( 1992). 77t« Feasibiliry of Using 
International Data to Set Achievement 
Levels for the National Assessment of 
Educational Progress (NAEP). Madison. 
Wis.: National Center for Research in 
Mathematical Sciences Education. 



Thomas A. Romberg is Sears Roebuck 
Foundation-Bascom Professor in Educa- 
tion and Director, National Center for 
Research in Mathematical Sciences 
Education, Wisconsin Center for Educa- 
tion Research, at the University of 
Wisconsin-Madison, 1025 W. Johnson 
St., Madison, Wl 53706, 



ERIC 



148 



Q. 
> 



a 



0) 

o 

0) 



u 
c 
<fl 

E 

w 

a. 

•a 

<u 

in 

n 
J2 
I • 
in 

•O <n 
w a> 

-S 00 

C o 

19 Q, 
<n _ 

° § 

DO 

= 

in J_ 



■2 § 
— o 

o >- 



E " 



•So 



v> 3 



i-S 



CO C 

.y .2 

c >- (0 

a> -£ 

D 
O 
w 

<0 



= _ t£ 



•a 

c 

"re 
c 
.9 

"♦3 

re 
c 



o 

0} 
w 
3 

re 
E 

V 
3 



•a 

c 
re 



ID CQ 

E S 

0) ^ 

Ea> 
oo 
^ eft 
a> >- 

■S « 
•a 

IE i 

00 ^ 

g.i 

Jl re 
H U 
3 
■D 



c 



S.9 

O 3 
tn 

.£ S 

? -2 

o z 



tn 

c 2 ^ 

Q. a> re 

0 » 

— tt) ■!£ 
a> tn u 

> 0) 0) 

« •£ E 

^ 00 a 

<»- c a> 

"O «n 

aj t ™ 
E S? 00 

0) 0) c 
"D 

«n _J 

1 ^1 
™^ 

g _c re 
0) 'Z c 

^ I E 
-tj O re 
< O. -D 



DO 
C 



re 



o 
5 



re 



3 
«n 

re 
E 



•a — 

o re 

JZ 0) 



i. re 
<n J- 



c 
o 



-o 

0) 



re 

J3 



c 
o 



o 
u 



c 
re 

tn 

C . 

o 

S E 

O tn 
tn 

tn OJ 

X tn 

C re 

CL E 

<u o 

tn ^ 

tn u 

0) re 
2 



c 

3 



-o 
c 
re 



>- 
re 

a 

X 

re 



re 
c 

3 

o 
u 
u 
re 

a> 
re 
tn 



2-s 
E| 

E Z 

O OJ 

>- - 

.E 00 

tn re 

y - 

1 - 
§. « 

? ? 

si 



^ 3 



8 



u 
</1 



0; 

tn w 
re re 



E 
0) 
-D 



•o 

E tn 

re ^ 

DO O 

c <« 

'53 tn 
00 

•- .E 

2 -5 

"5 re 

DO _, 

3 ™ 

ni tn 

E 5i 

0 0) 
re 

> 0) 

> ^ 

5 a; 

O) 00 

c re 

1 I 



-E-g 

S-g 

c 
«3 



a; ^ 
•£ O 

Ere 
o 

in DO 
in *~ 
W IV 

!« ■£ 

2^ 

tn "P 

s 5. 

to = 
-o ™ 

■£ q5 
O 

r a: 

£ re 

E i 



•5 E 

-51 

6 S 

u = 

tn V 

re 

" re 



re 

E 

OJ 
ro 

5 

c 

o 



re £! 

w o 

■i E 

tn C 

.y re 

1 ^ 

E « 

Q> C 

S ^ 

re ™ 



o 

o s 

Q. 2 

o o 

s. ^ 

o 

B S 



00 w 

£ •> 

in O „ 

y = « 

«" "§ 
o ™ O 



tn tn 
— Si 

15 a 



8. 



0) 
BO 

c 
-Si 

JZ 

U 
I- 



E 
E 

3 

to 

"re 

c 
o 

re 
Z 

V 



c — 

— Q> 

0 -= 

1 S 

a: c 
c re 

% = 

o S 

u _ 

« re 
c 

ty> o 

T- re 

— Z 

D. a; 
< ■£ 



II 

re o 

C Q. 



~ (J 

1 E 

2 re 
>. E 

3 -O 

E 



w re 
3 <u 

re 4. 

OJ O 

E ■£ 

3 

? m 

UJ 

to 



3 



S 2 i2 « 



re 

E .E 

^ :2 

re o 
Vi >- 

£ -g 

re V 



0 re 
■° $ 
2 in 

^ re 

re E 

O) (U 

•D re 

1 ^ 

-5 o 
■2.y 

re o 

"g > 

re 01 

tS ^ 

m re 

c ■£ 

0) " 



a' 

I I 
> X 

O I' 

E c 
E ^ 

.b I 



c c 

0) 



c 



0) — 
tn — 

c 

JZ 
0) " 



re 



00 



-o 

c ? 
re "i: 



00 a; 



•52 



.s re *-< 

E 



CO 

c 



00 a; 
E 

O 

8^ 

re 

•a re 

5J ? 
re 

3 Q» 

■° lE 



^ E 

I 
re _c 

>- " 
~ c 
re u- 

:^ 



B 




1 




>» 


« 




o 




>» 




CO 


CO 




u 


o 


a. 




V 


01 




<4-l 


IM 


0 




u 


(U 












u 


)-i 




c 


(0 




(S 


3 


cu 




0 


V 




c 


o 






•H 


&< 


• 


o 


o 


>» 




CO 


>> 


4-; 


•H 




E 








c 






V 








<u 


(S 




•o 










z 


01 


(8 




1 


1 






V 


O 








CO 


V 


o 


< 








V 


JS 


c 










(0 




V 


i-H 




th 




As 


>» 


ci 


na 


DC 




a 






C/3 


o 




» 




n 






•H 


« 






u 


fO 


<4-l 




c 




OO 


•H 


»l 


o 


(8 


o 


0) 


c 








Z 


u 


U 


•H 


(8 




>» 




00 


c 


)-i 


E 




E 


V 


c 


•H 


3 


0) 




V 




•H 


)-l 


0) 




J3 


•o 




J3 




(8 




00 


(S 




(fl 




V 


(S 


•H 


u 


IM 


(8 




s 


s 


)-l 


< 


O 


3 



re c 
j: 1- 



I 

s 

»- c 

/) c 

QJ fC 




° -s 

(J nj 
t:: 

|| 
i § 

QJ tn 
OJ V 

0) sr 



DO 
0) c 



0) cu 
00 



il2 

>- o u 

XI in g 
COO 

■£ U 

re 3 — 
re re 

re .£ .2 

t; T3 re 

^ ^ Z 

u 



c 

> 
c 
o 



in 

E cD E 

O c O 

"re £ m 

^ s 1 

m re J 

2 o) u 

— >~ o 

O 



o) o 

DO 1— 

c 

re 

X .- 

_ Q. 
re tn 

c c 

<u — 

E 

"D J2 

C 3 

v2 y 

o re 



S re 

o u 

C 3 

re OJ 

X o) 



^ c 

DO O 

o ~ 

^ OJ 

= tn 

OJ u 

tn re 

in C 

!n (U 

re 

= ^ re BD .O t 

TO r- 



QJ in 
^ 3 



0) 
t_ 

3 

t3 

3 



E <i) 

5 E 
ts 8 



c 
.o 

z 



y 8 

c 



X 

0 J2 

-a 

c 

01 ^ 
u ^ 

03 O 
X ^ 
00 

c 



in 



tn 0) „ 

^ tn o 

D C C 

tn 0) i2 

= -D 

2 2 

Q. rM 

tn ^ 

tr <n 

■w o £ o 



in -E 

.y 

E ^ 

^ S 



5 .E 



If 

re ™ 



_c 

TO tn 

O "W 

E 



c O 
o E 



" ir. -P 



O C 



X it: 
U -D 

,0) 



^ -D 
= "O 
flj re 

^ c 

tn OJ 

■£ X 



^ re 

O tn 

? 

OJ OJ 
X C 
tn 

^ 1 

I! 

y ^ 
— w 

.1 I 



OJ 

u 
c 

_ o 
re » 

o S }5 

.ill 

.y ^ re 

5^ s 

re "i; 
re 



3 
D. 



O 
O 
X 

u 



OJ 

> 

re 
c 

w 

a; 



o 
re 



IJ ^ 

So 



E :^ " 



"D TO 

re JZ 
u 

Qi tn 

-D S 



to y 



u 
_re 

a 

c 

o 

E 

E 

o 

u 



er|c 



Q. 



c v> n 
n> — c 

E ^ 
- o S 

Q. 



c 

o 



C S 
« bJ O 

5; .«! v2 



Q. Q» 

^ •£ 
>■£ 

2 -a 

o ™ 
j: - 

v> « 

1/1 ^ 

Q. — 
>. ™ 

£ 2 
2 > 

0, i 



.y i= — 



_ « >- 

EfO 



o w 
5. E 



re DO j; 

i 

2 o oj- 

V O V 



>r 5 

00 

.E (u 

1 = 
o o) 

< 

JZ 

00 

n> 

— ^ 3 
re JS 00 

E c - 

0) := 



re 



»n 0» .1, 
^ ^ .2 

J5 - -o 



0) c 

Z - o 
00 W 



O w> > 



u 

s .1 



c 

00 QJ QJ 



c 

V> (Jl 



c 
re 



re \. c 

-9 5. 

Sri s 

" i o 

:J 3 n 

o o 



c « 

re c 
a> .- 



I > 
« o 

0) u 

■z 2 
u 

0) 

re re 
.2? Co 



1 

^ .y 

a> o 

o re 
" c 

0) 0) 

a >-- 

O 01 

Q. -C 
Q. 3 
re 



0) 



O 



>- 



c c Q. 3 re ^ 



.i E 

= 5 



•i: ^ 0) _. 
y> fl) o i — 



5!| 

c 

— _c 



re QJ 



>- c 
-t; o 

X 



QJ 



u re 



01 vn 
re jc re 

re c re 



§1 
> 55 



c :s 



o ^ 

J= 3 

vi >- 
00 

re .E 



a> re 

JZ 



re 

rC C 
3 

E 

E 
o 
u 



c 
o 

•C W 3 

y 
c 

3 

E 

E 
o 
u 



0.2 

II 

E -u 

■£ IS 
re U 



00 . 

.£ S 

.s <-> 

IE " 

.- Q. 

a> Q. 

-c re 



0) 

o 
c 
re 
■c 
o 
o. 

E 



J2 

^ c 

1^ re 

J a; 

C v» 

re o 

111 C 



re 0) „ — 
•£ .§ 



re 



re .. 
1 - 



re V 
re 

-c -c 
re c 

5 £ 



d re 



CUv^ 

X o 

O) 

2 § 

00 tfi ~ 

Q. re 

3 c 
o _re 

00 Q. 
.- X 
.£ 

V 0) 

1 i 

o 2 

« "2 
a> c 
•i3 re 

o ■£ 

Q. O 



9^ A i 
o g c 

" g w 

~ .Si -o 
00 c 
0) o 
M >^ 
o »5 0) 

~ i5 ^ 
T3 ^ 

3 ^ "re 

0 .> ^ 

-grew 

QJ > 
V> w 
^ (J _^ 

!2 ^ « 

Ore 

^ trr 
I- <u , 

S > 

1 " I 



3 

■c 
o 

Q. 
O 

QJ 
N 



C C 

2 a; 



re 



- 5 
re QJ 
u c 

S .52 
1^ 

C 0) 

^ -B 
1 2 

_re 
'u 
O 

</> 
re 



in 

c o -5 

"" re 
E 



6 ^ 

CL c re 

O 0) *- ^ u< 

15 S = :£ 

.o O ™ re re 

i2 o. TS E E 



0) o 52 

o iS 

Q- c o> 

QJ *— <n 



QJ 



re 

JZ 

? 

Is 
-a 
</> „ 

|i 

™ re 

• « u 

t/i ^ 

o >- 

-- 0) 

re ■£ 

QJ re 

^1 
re > 

E 2 

_>- -D 

a w 

Q. o 
re Q. 

& 

v> O 
3 -D 

2 S 
«> 

Q) >- 



QJ O 

> -D 

re ^ VI 
3 

O « ^ 

-g O 2 

«« Q) 



QJ 
JZ 



(f .E 00 

.2 op g, 

o -E ™ 

2 ^ 



■ J- w 

•c •= c 

QJ O 
Q. > U5 

X re 
« & u 

QJ w 0) 

JZ v a. 
c X 

O- C Q) 

O Q) 

— "O 

— -D Q) 

m 3 n, 

s s !s 

S2 S.-D 
0) •- c 
u re 

re *- flJ 
0) re tj 
•= Q. c 
0) 

Q) O T5 
^ 

S 4 c 

Q) C V 

F Q> 



■~ — E 

1, 

Qj ^ 

s •_ c 
O o re 



E .2 S 



6 -5 

u — 

0) -t: -a 

-o -5 S 

^ re 

3 0, QJ 

I/) QJ 



«n 
0) 

VI 

3 
O 

u 

VI 

u 

re 
E 

O) 
.£ 

re 
E 

QJ 
w 
3 

.3 



• 



Cvl 



OJ 



GO 



o 
o 



O O 

Q. V> 

™ o 

QJ DO 
3 

o -a 

J re 

II 

QJ V) 

*U 5b 

^ 2 

v> 

C -_ 

E 1 
!;! re 

Q) C 



T5 T5 
Q) C 

c re 

> V) 

§ J 

>- re 

— 0) 
c " 

§ 52 
C7 2 
<« re 



v> 



v> ~a 



a> V) 

~ VI 

re re 

" E 

V) O 

"i_ u 

QJ C 

E 

o 



^ 2 
.E 



c c 



T5 
QJ 



o 

O v> 
c « 



QJ 



V) I- c 

re 3 o 

v~ o ~ o 

Q) a: 

°- = 

>- v> T3 

— re 0) 

QJ QJ V> 

i E .y 

2 I 

re .2 75 

X ^ (— 

0) o. t 



(/» TO 
Q) P 
Q. ^ 

if 

a o 

O) 00 

■£ 00 

0! -i 

re 'w 

S! ? 
u 

,0 re 

^ E 

v> 

re 



QJ v~. 

> o 
o i_ 
O ^ 

^ I 
= 1 

00 VI 

E " 

|2 a; 



re 
VI 

j: ? 

Q. 

X 0) 

(u re 

re _3 

re •— 
-£ JZ 

II 

u 

3 „- 

V) v> 

.E IS 

VI QJ 
U VI 
— QJ 

re ~ 



a. 
E 



11" 

ii 



3 
_C 

C 

o 
u 



0, 0, 

JTL JZ 



" c y= 

5 re o" 

re c 

Q) 5 

E -D 

>~ id c 

=: £! re 

S 2 " 

il v> 



JZ c 



E »- .2 



re 

u 
o. 

3 
O 
DO 
00 

c 



re p 
E 

•£ o re 

~ '~ \ri 

DO £; 



re 

U c 

3 — 

"D V) 

Q) -51 

JO SJ 



i_ re 

3 w 



3 5 
XI o 



o -o 
re 

c 
re 



re " OJ U 
E J i Z 



^ E a: ■£ 

>- ~ n ni 



re 
T5 re 



C 
QJ 

E 



U.' _ — 

V. ^ IS ^ 

QJ r- — 
Q. .- Q) ^ 

2 ^ i o .52 



O ^ 
Q. .3 

v> 5s 

QJ _l£f 

— re 



T5 



V. ^ 
re 



re 3i.s;| 

JZ -O 
00 

Q) .E 
VI 00 

— C 

1| 
T3 JZ 

re u 

re c 

•n QJ 
> 



v> •- g 

i3 .E o 

s; |; 5 

re > >" 
a. 



c 

QJ 
> 
QJ 

T5 
C 

re 



DO — 
C 00 

.E 
Q. re 



•3 C 



DO O 



QJ 
re 

QJ 



-C .E QJ 

U 00 ^ 

V) 00 QJ 

eii £ -o 

JZ ^ ^ 

0) o .t; 

K J3 O) 



QJ 



c 

QJ 
>< 
QJ 

"D 

0) 

re 

X) 

QJ 
"D 

O. 
3 
O 



00 

c 



QJ 
U 

>-. 

QJ 
JZ 



I 



■s % 



i: Q) 



VI 



w 2 

CQ 

UJ 
ID 



00 



QJ Q 



U o 

q; Z 



u 

a; 

QJ -D 

v> 

re 3 Jr 

•5. 2 „ 
O I re 
"i; 00 re 



I- QJ VI 

™ c 

.2 -c o 

!r 

5! V, re 
■= p 

U Q) t 

0) ^ v2 
i § .E 

^ Q. QJ 

a; QJ ^ 
"5 •£ D. 



.E 

" Q) 
_ QJ O 

« o 

Q, " Q. 

>- "O 

O QJ O 

00 f- 

2 ol o 
- E "6 

3 



3; xi 



u 

QJ _Q) 

C >" 
O -D 

" c 



QJ VI 

'Z. 3 



v> .2 r! 



0) 

u 

C Q) 

o; vi 



"D re 
c 

QJ 

<u ■£ 

C7 JC 

QJ w QJ 

v; Q. 

c t re 
u VI 



QJ 
v» 
O 
JZ 



QJ .. 

c- aj 
aj - £ 

3 



QJ 



OJ 



QJ 
X) 



U 
QJ 



OJ 



3 
O 



O T 

'w QJ re 
™ 

v> QJ QJ 

v> 21 



v> 

re 



t 

re 
5 



o. 

a 

£ 5 

c c c 

w o — 

3 U T5 

-° - 3 

^ ™ o 

« 3 V. 

E — ^ 

s ^ " 

o 3 2 

E ^ Q) 
_ _re 

^ 3 ^ 

^ u o 

"qj >- 

™ E 

^ Q. ^ 

:=| i 

O. P re 

V> -77 

« IT Q. 



£ 

re ^ 
■K o 

v> *i 

re 

^ Q> 

u 

3 

v» re 
re 



0) 



£ re E 
re j: re 

-F I 
_re X) ^ 

■c t. ™ 

re Q. c 

E ^.E 

^ E a) 
2 o .N 

t QJ V) 

00 2 



>- u E 

QJ w QJ 



Q) • 

1— -C 

w (J 

re 

.E S 

E 

I § 

o »- 
5 VI 

E ™ 

Q. ^ 



QJ 

> ^ 



E 

> 
o 

E 



OJ 



-y 

E E 
Qj a! 



re re 



ERIC 



c 
n 

tx. 

E 



E 



.£ Z 



0) 

c 

o 



Q. 



IS Q. E 



0) 



JC 
u 



Q. 
>• 

O 

9 

Q. 



w 

5^ Q» O 

g C U 

D C W 

— o 2 

.E S 

« 2P «« 

P *= o 

5 «" ;s 

2 a; 

c 2 E 

n tz s 

-= 0) ^ 



2i ^ 



c 

(0 

f f - 



■c o 

E «^ 

00 "tn 
O 



0) -P. 
T5 
C 



u 

(0 

o 



Q. 

a 

7 to 
00 oj 

q3 > 



2 _c -t- 
V- -o •£ o - 



O t 



c 

o 

E 
2 

00 

c 

c 

f8 



<— o ^ 
•»= a> .2 

c « 

E i iS 



i| 

> vt 



<« ■- 
> 2 
"S 2 

QJ 1/1 
OJ 



00 o 



OJ 



E^ 



OJ X 
OJ a^ 



c 

OJ 

W I " 



OJ re 

c ^ 
c 

2! S 



U 

c 

a; 

< 



Q. 

re 

o 



C 

o 

oi 
Q. 

OJ 
J3 



•i ! 

re 

I E 

re 3 

'-I i: 

3 tS 

-o c 

QJ — 

^ E 



5 " 3 

^ C 3 
Jr O) T3 
O E a, 

* Q. C 

O o — 
-f -S Si 

0) -D w 

o - S 
■B ^2 

JC ^ OJ 

o .5 ? 



9) 

r- in 

■S «« 

re re 

E ^ 

-o c 

OJ re 

E a 

OJ <n 

U 0) 

C ' 

S o 

^ in 

=: 0) 

re Q. 

c o 

.2 -S 

<n > 

<n QJ 



0) O 



in Of 

3 re 

^ E 

•5 ^ 

C m 

K «> 

P o 

a> 

<n -C 

— U 

OJ re 



w 

in 
_«J 

re 

E 

V 

re 

E 



re 
T5 
c 
re 



0) 

5 



OJ 

-o 
re 

V 
w 

>- 
c 
re 



u 



bo 
.C 

i: 
2f 

I 

E 
o 



C 
.0 

re 
c 

OJ 



a. 

0. 
re 

OJ 

c 



OJ 



re 



c 

0) 

E 

Q. 
O 

- ^ 

I- 

re 



re 
O 
00 

-o 

re 

Mi 



OJ 



-C: 



re 



re 3 -i: U 



2; g re 

^ 1 

^ T3 in 

— QJ ^ 

re ^ Q) 

O) S g 



U OI 

= E 



•n * 5f! 

<U ^ OJ 

^ re " 

2 v> >- 

o £ -o 

re (o 

(P E 0) 

m c 

OJ >. c 



00 re 

- -I I 

O 

o ^ n3 

- c "o 

c P ^ 



in — - ^ 



U 
3 



in 



O) 

re 

T3 

c 
re 

E 



OJ 
in 
OJ 

00 O 
3 

in m 
(U 

in 



J5 00 VJ c 

E 



C 

o 

OJ 



re 

2 8 

2- <-> 

re in 



8 5 
in .a 

in 

re — 

u 



E 



ft) s 



-O in = <- ^ 

- ^ e To 

1 1 



•£ .y e 
c "o 2 

— D. re 

-5. 

-i J .y 

in "O re 
00 re c 

C OJ 

re ^ J 

"1 i 

•£ >> re 
-o £ * 

O in 

w ^ ^ 
Si! ■? «» • 
^ * E 

" J5 c o 
^ 2-S E 



o ^ 

in 
in 

0 C 

U 00 

^ i 
to ^ 
c 

u a; 

^ -£ 

in 

q5 re 

^ lA 

I- O 

■£ 

OJ ti; 

u 3 

c — 

0) w 

1 ° 

re -D 

_ C 

TJ ro 



a5 "re *^ c i_ 

< ^ 2 a> O) fc 

S re g 2 « 

S c K tn re 



£ 0; 

8 ^ 
■5 1 

in 3 

re 

c re 

E-i 

00 g 

.E a, 

S 2 



c 

O in 

•z: O) 

re a. 

s >■ 

c. Q 



re _ 
E « 

re 8 



fe. E 

o a' 

O m 

C. " 
J" 

•- £ 
,_• c 
£ E 

.i! 

re re 



IS 2 J 



D.J2 

c iS 
re -c 



— O 
re I" 



in 

O ^ 

= -5 
QJ re 



re 

O in 

00 _a; 

w C 
re E 

•£ a-' 
re 

^1 
in 

c _ 



re 



OJ 



(J lu 

s. > 

X a; 
0) ^ 



O 
•p 



bo 



OJ o 



> 
U 



re 

E c 



c 

OJ 

E 

_ 3 

C OJ ^ 

■: .E 

o OJ ^ 

-O >• ^ 
S Si " 



in in 



OJ OJ 

-£ JS 
.E o 



U 



re -^s ^ 0) 



c 

81 

S 2 S " 
o o 

2-§ 
a 3 
0, o -o 



re 00 
_ o 

re o 



OJ 



2| 



OJ ^ 



l~ 'E tn 

-I = 

CO 



CL 

E 

re "D 

X OJ 

c 

re > 

QJ OJ in 

2 2 "E. 

^ 2 a) 
o ^ ,2 



>- 00 

^ E 
^ •= 
S3 J 
£ o) 
S ■£ 

re >^ 
^ O 

^ I 

•i i 

II 

U O 

<u -o 

'u OJ 
in 
O 
Q. 



S °-2 

re Q. -2 

c 5" " 

OJ ^ 

E 

^ -c 

D. 3 

Ew 

— <u 

o 5 



i_ OJ 



in 

C <^ 

o — 
re 

ro O 
w 00 
OJ 

"O CO 



OJ D. — 



OJ 



o 

o c 

-o =^ 
a; T> 
™ re 

QJ 

2- QJ 
re :0 

.ti -O 

oj 

5 



t . 

O OJ 
" in 

a) ■£ 

OJ ,_ 

2 
2 3 

DO 



o 

o 
c 

QJ 
00 

c 
re 

u 



QJ 

i a; 

00 in 

o 5 



01 



o c 

E £; 

2 3 

CL U 




a 



-o e 

Si V 

c "2 
% 8. 

v> O 

2 g 

in n> 
in 

m ^ 
«n 

4^ « 
•il 

CO ^ 

S E 
c .« 

II 

o 



CO «« 

^ •£ == 

w j« 
re 

W > 
3 > O 

• ^ DO 

•5 ~ "o 

■obJ2 *> 
c >" S 

o «» 

•- o y 



s s 



5 = 1 



£ 75 

rt a> ™ 

*^ ^ E 

i/> o 1^ 

^ o 

a, — 

S ^ a; 

•><!> — 

^ ^ 0) 

C ^ 3 

a? g ^ 

2 ^ _ 

IE -2 S! 

u .op •£ 

«£ ^ J 

T3 £ O 

ro t3 *n 



P 

c S 
•S S o. 
•n S « 

^ °-lS 
re ^ 
O S «? 

>" S s 

o u •a 
o. .-JS 

c = 

" 1 
? = c 

.2 o 00 



S " 

Q. m 

S" in 



O 



0) s 



_ 2 op 

re c 

O) fi => 

tn -C r" 

DO k 

3 2 

w i ■£ 

2 "5 s 



0) 



^ «n 

t- "> 

U 2 
re 



2 - i- C re S 12 

2 Jc '5 S 5J °- « 
-o 

rt c c 

>..2 " 

5 a o 

S -D -i: 



i; o 

in 

3. o 

= ? -n 

Q. "So 

^ re UJ 




ID 



5 
5 



o 
o 

CL 



ERIC 



■re 5 

DO C 
m " 

V 
o E 

tn w 

<u c 
■q. "> 

E E 
re 

X <i> 
JC 

m *^ 

re c 
™ ^ 

— XI 

-8:2 

3 

00 ^ 

E c 
G .2 

're i 
E E 

^ .E 

E £ 



o c 

— o 
tS u 

o 

< « 

QJ 

in 
*n 

— re 5 

;^ E f 
> s o 

^ ^ 

rt) BO ft* 

° ^ 

3 T5 52 
i: c re 

tn — 4) 
C >" Q. 

0 u 0) 

<n a; i2 
re " m 

01 (I) 01 

re u c 

I § 
QJ K u 



3 
■D 

T3 



re 

J3 



re 
u 

3 

<2 QJ 
QJ (_> 



re QJ 

a, i 
.o 

o ° 

^ g 

00 " 

C QJ 



0) 

<J 

Xj 
2 

Q. 

E 
o 
o 



o <u 

>■ Q. 



w "±2 tn 

3 ns 

m 00 73 

QJ •- O 

5 u r; 



T3 
C 



QJ 



>- ^ E 



T3 O 



c 
o 

Q. 



QJ 
U 

c 

re ™ 
I- 

C >n 
< O 



i2 3 .9 



QJ 

u 
re 

H) re 



O 5 

_ 3 

■o 

QJ 



QJ <U 

a ij 



re 

E 

QJ 



:s o 3 
re w o 

U Q. vn 



QJ 



SI 
oj " 
•£ •- 

t — ^ 



T3 ™ 

C 
00 

•o; E 

QJ QJ 

-o ^ 



c c 

(U (U 

> >- 

o) IE 



i> 2 



^= -5; I- a; - 



QJ 

y OJ 

re 3 

2; ^ 

re 

E - 

t/1 

a, — 

>- c 

3 re 
o 

re 



o J2 



QJ E 

$ DO 

%n C 

i -5 

E o 

o „ 

o i 

C 0) 

S! T3 

? § 
DO 

U o) ^ 

— -D 
O 

2 P 



re 
u 



re ro 

2f2 

JZ 

u m 

re QJ 

Si 'do 



CL • - 

c 

3 

o 

CL 
Q. 
O 

c 
re 

E 

QJ 



C 

<U QJ 

2 j= > 



*— re 
c 
o 



3 

re 



in 



QJ 



C in 
•- (U 

QJ 

00 y> 



_QJ 

"re 



E — o 



u a 

C >n 

E 

in 



-o 

C 

re 



>- ^ 

TO re 



C 
ro 

VI D. 

E „ 

QJ re 



O 



QJ s: o 
E c. 



o 
E 

QJ ^ 2 

QJ O 
O X! 

.£ £ 

00 > 

a> QJ 
-o V, 
£ E 



Q. 

E 
o 
u 
u 
re 



X 



c 
o 
E 

QJ 

-a 



— re 
>- E 



a. 

o 
o 

a; 



o 
o 

CL 
QJ ^ 

E -S 

CO u 



_- c 
re — 
E XI 
00 5i 

'C re 
O QJ 

CL 
0) CL 

i: TO 



_2 re 

m "~ 

re 

*^ tn 

QJ re 
X 

— QJ 

w X 
O 



QJ •£ 
QJ 

•C c 

X 't. 

o 



X u 
Q) OI 



c 
o 
u 



o ™ 



QJ 

o E 
o 

m m 

00 

c 

= Q) 



o T 
CL 
>- 

X re 

_>--D 

QJ re 
00 X 

re QJ 
— > 
re ^ 



3 

o 



Q. <u 

00 re 
c X 



m m • — 



-5^ 



3 o 



> 

a 
E 



S I 

QJ 
X 
QJ 



r c 

re — a; 
CL _QJ X 



x > 

QJ > 

£ S! 
c 12 

'S! ^ 
1^ 

Q. XI 
^ ™ 

00 DO 

•S X 

c» *^ 

= 3 

^ J 
ti 

E ^ 

QJ 

m • — 
^ T3 

re re 



S re 



2 ? 

I- re 
U 

Z 12 

■£ re 
X 

X 

^ ? 

X o 



a 

E 
o 
u 
u 

I 

05 



re 

c to 

(J QJ 



QJ >; 

QJ re 
Q^ 

Q) 

u 



U O 
QJ 

c ^ 

re *- 



X 

u 

3 



C j«: 

OJ V) 

> 2 

o «u 

-o X 

O Q. 
QJ 

ul QJ 
QJ 

"O o 

re '-' 

»— m 

00 re 



_ 5 E 
O re 

O re «n 
"D v 

o -S! 
^ — a> 

oof 
^ ^ § 



O ^ 

.^0 2 



_ E 



QJ 



re O 



n to X w 



^ in 

u 

QJ a; <5 

Q. 2 

o =^ c 

— -DO 
.E 3 = 

o o o; 

Q. J 5 



re E 
^ o 

0 S 

-I 

in_ <U 

01 re 

2 J2 

DO «n 

^ 2 
II 

\1 \ 
-S 5 ^ 
2 c 
w c c 

O re t 

— in c 
in >- 

O a> 

m J3 D 

Qj n n 

u e. 

U V C 

>e j3 a. 



a 
3 



£ c 
-c > 

o 

00 

CO ^ 
3 .Sf> 

E ^ 



if J 

« £ J 

P £ a> 



a 
o 

in 

•o 
o 
a 

X 

D. a; — 



1 i o 

^ > c 

£""T3 -O 

> « g 

oox 
c «« 



O 0) 

> c 



re O 
4j tn 

re 

>1 

0) o 

c a> 

D-D 

re re 

i_ I— 

00 00 



3 

s! £ 

•£ -- 

O (A 

IS 

in 

c « 

a> re 
CL E 



- a 

o £ 

re c 

o Cl> 

C i. 

.S2 ° 

tt) J; 

■£ — 

in "5 

0) 1- 

-o « 

re 2 

00 o 



0) V 
3 O 



w 

o 



3 
O 

I- 



0) 
xn 
V 

re 

a> in 

> 

Si ^ 

c O 
O w 

- •=6 

re o 

|i 

2 c 

O re 
a c 



1-5 
or E 

> -D 
C 

^ >. 

o -5 
« >- 

re «^ 

c .!S 

in £r 
0) 



" F. 
It I dj 

3 



% in 

I 
O) CL 

o •£ 

3r o 

CD w 
in a 

re ^ 
2 3 

E ^; 



2.^ E 

re 



5 ^ 



c 



re -t: 



0) C) 

Jl - 



0) 



*- in 

»- re 

QJ — ' 

QJ *^ 

<U op 

2 1 

-D a. 

S.i 

^ re 



u 

E re o 
_ O j: 

■D DO 

tt* = 

c 5J o 

re *^ ^ 
m r- 

— E O 

a> 2 '> 
J or o 

a = 

— c 

V 
T5 
3 



O ^ 



u 
E 

c 



QJ 



E 

o 
o 



3 -a 

U 01 
<n C 

2 ^ 
re 

o. Q 
m re 

II 



c 
w 

E 



= -D 

re c 

— Si 
•S X 
•£ *» 

» c 
._ re 

Q. O 
3 O 

~ 

CL 

o X 

0) 

in 

0) O 
D. •» 



0) 00 



re 0) 



^ -1— — 1 



? E 



u c 

Q. c ui 
~ a> 

s= ^ 

a* J 2 
S S 2 

^ 0) Q. 

(= -C (U 
' u m 

re a> 



-D Q- 

0) E 

c re 

in a> 

£ o 

CL C 

= ^ 

re 

-E ^ 
00 -S= 

•C op 

° s 

in — 

re re 

in X 



-c Si 

.s; o 



00 

? ? 
CL a 
O re 

Q. E 

V 0) 
00 JZ 

re 



re re 

" >- 

j: — 

U in 
1 

•E 3 

^ o 

w u 
re 

*-* in 

» c 

00 ni 

.E -D 

C 3 



re o 

00 $ 

c — 

-t; u 

T5 a, 



c > 
.2 00 

m re 

3 m 

cr — 
re re ^D 

r- 

O •£ T5 
LO "D 



-re a 



E c 2 

5 — <u 



£ 2? c g 



J= «" 
.op O) 



3 _^ <U 




O) re S 

> X 3 o 
a> oi D^ w 



00 



• 




ERIC 



J3 o 
11 



11 

C «A 

o a 

§ \ 



c 
u 

3 



•£ o 



£ w ■£ ^ £ 



E - 
a, .!2 

Q. 
in 

J) « 

«fl *" 

C >- 

-2 5 
2 E 
™ .£ 

en 



ll 

IJ 

s 1 

E 
E 
o 
(J 

c 
o 

n 

u 

3 
■D 



0) 

5 



E 
>< 

OJ 

c 

OJ 



to 
IS 

§1 



E 
E 



« >. 

« §1 
a> £ 

S o 
t -o 
_ <u 

c 

01 



01 

c 

01 c 
00 — 
«n 

O) — 



sp o i s 

'C C «n (U 

3 o) re j:; 

«o c — ^ 

<U ~ 0) re 



c •— 

o ^ 
■zz « 

o. .2 
•E a. 

:2 

o > 

il 

■S! 
re 2 

c 

0) o 
°- C 

.E S 

re -O 

E 2 

0) o 

•B I 

00 0) 



O JC 

o> °- 

O 01 



«A 

c 

c -2 

O) u 
(A m 
O S 



•S }5 



o 

u. 

re 



c 
VI O 

(J wt 
^ & 

M C 

.2 r 

IS <u 
re u 

« >. 

o I 
u ' 

c 

00 re 
-D ■£ 

01 

S3 



£ 55 
5 o 



JO J5 

E = 

3 JC 
C ^ 

■5.1 

•e o> 

re " 
a 



o a, 

^ re 

•£ VI 

"5 o 

VI '.S 

S o> 

O VI 
Q. J3 

|5 
._ W 

n2 0) 



O) S 
f a; 



a; 

I"? 
> re 

S •= "> J= 
« re 

JC o 

II 



Q. 51. re 
00 5 ? 

ego 
^ o c 

re ^ c 

♦J 0) — 
•« 00 0) 

o) " S; 
r; V) re 

S « 

2 E ^ 

3 «^ >• 

8 5 2 
VI ^ 2 

S c 
™ .E js 

0) a> 
-c c 



3 

u 

C 
re 
Q. 

OJ 

VI 



O _ 
1- U -C 
O 3 



>- -D 

a; c 
■£ o 
IJ 

VI OJ 

re VI 



0) 

> _ 

3 VI 

/- O) 

a; $ 



c 
o 



IJ 
OJ 



re 00 m 

^ o " 

^ Ql 3 

i ^-E 

VI 5 ^ 

.E ° S 
VI re 

O — O) 

(J — c 

VI — C 



3 s 85 
I 2 -ti 

** O) >. 



C .52 



T3 S, 
01 



re 

V) I C 
O) ' O) 

c ^ - 
o 



J5 
p 



U 



•e re ^ 



5 c 
Q. re 



o c 

u O) 

« E 

X> OJ 

VI 

"D VI 

— re 

° s 

5 Z 

VI O 

O; ^ 

c = 

E 

w> O 

C <-> 

O) — 

T3 re 

:= r. 

J=. o 

U Q. 

5 i 

° c 
re 



VI 0; OJ 

oi •£ .2 

^ .£ 5 

c c 

0 S 

^ 3 •= 

k- -o «« 

re OJ 

> «! u 

re o c 

^ re «» • 

01 OJ w }^ 



OJ 



OJ S3 

o3 S 
J= o 

= 2 

.1 ^ 

00 a, 

V. u 

.2 I 

^ j= o> I 

"J .ep o) 



0 o 

-u 

S • 

re OJ 

01 00 

VI re 

OJ U 

re ^ 



OJ "S-S 
- =1 -a 

.2 "5 



o 
c -S 
.2 o 
% * 
.2 0. 



> OJ 

2 2 I g 



00 



.£ w £ 



(J 
c 



DO 

're 

'c 
o 
(J 

c 
.2 

<u 



= a; 



OJ 



re VI 
o> 

0) (J 

•£ 3 

^ o 

c 
re 

> 



-f. u p -j: 



!- o ™ — u. 

" 2 5 c 
re -s c re 



OJ 

E - 

O OJ 

2 

c 

Sf 



3 

o 

E 
o. 



re o 

E o 

0 ._ 

U 01 

£■1 
|g 

O ™ 
(J 1- 

o. «2 

1 i 

J= OJ 
— -D 

c 5 

VI 

iS o 

-D 

JZ 01 

<-' m 

™ -S 
OJ 

r. op 



|8 

< 3 
O 
S3 
re 



OJ 
Q. 

OJ .y 
-•I 

-D 2 

« 00 

o .£ 



o 

u 



OJ 



re 

5 -D 



OJ 



©•:=>. 

-C ?, re 

Vi — 

OJ ^ o 
"5 «» o 
2 — ^ 

tn U 
<3 VI 

m 

OJ 2 — 

0) TJ r: 

^ OJ — 

— re 

w ^ > 



Or u 

OJ ■•S - 

IJ 2 c 

c t 

OJ OJ 



-a 

3 
ro 



C 

E 

E 
o 

u 



re -— 



re 

E 

-D 
OJ 

c "2 2 
1^ ? 

— OJ C 

OJ > £ 
^ re O 



CO 

bo ^ 

o 



p 



OJ .2 

2 I 

00 3 

s: °'-d 

e OJ OJ 

3 a. 
o Q. 

OJ o 

0 C T3 
■~" "ul <£ 

«f ^- 2 

y OJ 5 
re OJ 

E J c 

OJ o o 
■£ I 

E o; § 

C OJ 

"5 flJ — 

1 1 2 

OJ = 

e > ^ 

re OJ 3 

-2-1 

re >— > 

c re 5 

OJ — u 

? ^ -5 



VI .2 
re j: 

5 5 



re 



OJ 



* E 

■ t- 

c o 

s _^ 

-D I- 

-D r 

<« VI 

re re 
^ c 

OJ J 

2 
o 

re 

"5 5 

1 g 

OJ V 

c - 
re 



0 .i: 

01 •£ 

00 J. 

re — 

§ ^ 

2 ^ 

3 

re o 

c X) 



-D -D 

C C 

re re 

"S I 

f s 

OJ 01 

o 

>■ £ 

0) OJ 

E ^ 

OJ a; 

5 -° 



UJ _> 

o 

3 OJ 



J3 2 

I i 

OJ 



■D 

OJ 



VI 



B -5 

>i= c 

1- o 

OJ 

^ OJ 

re ™ 

Si E 

■5 5P 

o E 



VI OJ 

OJ p 
in 

VI OJ 

re OJ 

c r 

re — 

VI OJ 



ro 
IJ 

>- 
c 
re 



c 
o 



Q. OJ 
3 J= 
O — 



01 ro 
5 E 



-a 

OJ 



S c 

Q. o 

re VI 

*0 re 

2 o 

re ,_ 

^ E 

VI 3 

2 c 



re c 



°° ^ -E t 

o E 

it: 00 

c c 



OJ aj 
rC > 



^'t> tvi^oj*;; 

^•"t; PiroOJjZo 
rereD. o 



c :i; 
O X 
•ob 

OJ "O 
*- OJ 
OJ 

3 

cr 



re 

E o 

<« -o 

O 01 

VI 5 

^ ^ 

OJ re 

■o 

2 "D 
OJ 

5 i 

3 VI 

> " 

:S OJ 



.h O 1= — 



^ 2 -Q 

3 

o "E 

'-' 3 



OJ 2 



re "o 
C 3 
C IJ 



SO? 
re aj 



^ 2 
OJ 

■£ re 
00 -n 

•i I 

re ^ 
re £ 

Q. ^ 
Si ™ 

^ VI 



OJ 
S3 



OJ 
■D 
_3 

U 

c 



VI 



DO 

o 

OJ ^ 



VI 

is re 

ro V 

-c 03 

3 U 

vi ro 

VI OJ 

Q. 

ro aj 

OJ 

Q. o 

is OJ 



ro E O 



ro 
E 

_0J 

S3 
O 



O 
O 

OJ 
S3 

O 



E 
o 

u- 



q5 

c re 

:= > 

3 C 

o re 
5 



OJ 



2 6 
aj ^ 

00 ^ 

00 M 

3 re 

VI 



aj 

op g 

VI S 

■g re 
c 

OJ E 

VI 

OJ 

^■g 

.2 « 

aj 

« 00 

i2 2 

c re 

OJ VI 

aj .!2 

C aJ 

£ Sj 

« -B 

Z -2 

^ -2 

^ OJ 

2 •£ 

re *^ 

UJ 1- 

OJ 

< 



00 

c 

-D 00 
OJ I' w 

? ^ ° 

2 ^ E 
^ i ^ 

-15 c 



re 

VI 

£ "5) .y 

. O ro 
v> ^ Q. 

"o "re ™ 

1 1 

^ 1 

01 00 2 

OJ VI 

re «» 

2 C TJ 
o c 

OJ VI aj 

c «^ o 
o c 

E-S3 
o 2 o 

v> C 



>tr U 00 

O C 

O OJ 3 

Q- VI VI 

•5 2 

V) -o 

OJ VI c 

-a £ 3 

™ ^ 2 

VI -J 

2 Z s 

•£ V — 

OJ re 

VI u 

£ re -i; 

._ (J re 



s ^ I 

.- _c E 

^ ^vi 

v< . c 

c 



OJ 

c 

0) 
I/) 

a 

OJ 

o 



QJ 



■z: ^s: 



o 

V> ^ f- 

VI -C 

S u * 

!5 re c 



re 



QJ . — 



3 



I ^ 
OJ 

VI re 

w 

3 TJ 

O re 
u -c 

OJ 

o > 
re 



fc- r— QJ 

^ I £ :£ 

■= P Q. 

O v; re QJ 

t. VI Q. u 

" vj 2 

^ OJ Q. 

U C OJ OJ 
*" 

^ ^ .o 
^ ^; 

OJ 

re -B 



a o 
c 
o 
c 
g 

u 

OJ 

VI 

re ~ £ 



OJ 

^ ■£ "4 .SO 



OJ 

OJ re 
VI re 

^ E 

S4- OJ 

00 is 
2 c 

o E 



•±= p -p ^ 



VI 00 "J 

ni 

P U ^ 

8 -O w 
^ -D O 
X QJ VI 

3 OJ 
VI Q. 
VI VI 

re re 



re £ 
u 

ll 

I -2 



OJ TJ 

E § 

— lO 
3 

v^ 



^ 2 § 

re OJ 

X =^ 

OJ 



-s 

c 
re 



VI ~ > ^ 
— re c 
i2 c — T3 

OJ iZ .£ g 
(J 

OJ 



OJ 

o _ 

p - o 

VI w *^ 

i_ 01 — 

O £ X OJ 

OJ o OJ 

C Q. 0> rj 

° « -£ 5 

O 3 

re ^ 

IJ .r 



> 



t) o C - 

?| ° £ 
^ ^ 2 



if ^ ^ ^ .s; 

i QJ ^ w VI ^ VI 



f 8 

CD 



I 

•I s 

«2 S 

o o 

91 

M & 

° "o 

3 C ■— 

& '- (U 

I 2^ 

s g 

3 2 

« « 

Q. tn 

(XI u 

«:e 

1 2 

.2 ^ 
n 



O ' C 

— E «o 
•Co 

w w re 
00 u 

-E 

^ X C 
O oj 01 

y .!£ ^ 

" re « 

- ■ 

re >• ^ 
v-s re 

re 

X 
0) 



V n -is 



0) fs -s 

"> P Q. 

§.-§ I 

^ 2 = 

3 ''I 

«ii re 

5 2 B 
.i S-o 



^ 2 

Q. 

9- <u 

re j: 

1 2 

C -D 

o Si 

•5 ™ 
re 11 

|i 

re ^ 

C « 

5^ 

vi re 
u 

-- 0) 

E 2 
re i- 



«« c c 
cere 

O 3 (A 
~ in ii: 

c re •n 

E 

E 
o 
u 



2 I 

re 

O 



re re 



0) £ 



re 



0) 

Q. 

Q. c 

E " 
X o 

.5 I 



o) c 
re S> 



E 
(/) 
E 



OJ 

c 

o 



E a> 
o t 



Is 
^ re 



in C 

re 

r- X 

<U 



0) (V 

■5 S 



•u ^ JS — ^ 



c ^ 
0) u 
-D » 



a S 

c c 

I/I 

2 8 

£! E 

c 0» 

— j: 

ni re 

ffl 0) 



U JZ 

0) J 

z ° 

re 00 



01 00 
-C C 



0) 

E I 

^ I 
re .= 

re 

o := 

S! I 

^ 3 

in ■— 

re 



a •£ is .y 



t .E — 



u 

w 0) 

01 



2 C 

0) I>- 

<n It; 

O U 

Si -D 

O *rt 

C C 

< a 

in 



u 

CL ~; 

in 6 

re £ 

•= re 



I/I fO 

1.C 

= -D 

51)^ 

re 

_ in 

i 

._ re 

"I 

in 

0) O 

m 

0) 

to .t^ 

— ' >- 

<n — 
re re 



E 



in -n 

iS C 
re 

s — 

"re 
o u 



3 

u u 
re 

in Q. 

— C 
in o; 
re "D 



3 
U 

-D 
0) 
in 
re 

S3 



« 
C 



X re 

0) JZ 

§ i 

u o 

c 1 
■~ cr 

c „ 
0) 

re 43 

« o 

It 

-° E 

re O 

n *w 
U 

in -js 
i i 

01 o; 
.^^ 



c S 

2 a. 

X >• 

" o 

•£ 2 

in Q- 

■~ 01 

n 

>--5 



■£ o 

o ^ 



•2 

0) o 

£ c 
^ re 



in 
in 



3 "O 

•E °° 
a-E 

a> o 

in - 



$ 2 

-i 1 

01 o 

N o. 2 



= .r — *^ " 



n IE 

3 J 

S 2 



V tn 
^ =1 

re 

u 



o o -6 

■r, '•s i 



re 

_3 

re 

Uj 

T3 
C 

re 

s 

"a 
o 



-a 

in ^3 

"K c 

O) 
DO 

3 « 

in ^ 

*" re 

0) JZ 

t— *-» 

IZ 2 

. o 

in W 

re 



*- ."go 



E 

0) 



z 



5 £ 



D. •£ 
E o> 

0) « 

E JO 

1^ 

£ u 

3 vn 
00 

J= ^ 

.!£ 2 
u 

-O => 

*-* in 
U 3 

H 2 
£ & 

y re 

■> «« 
re s 



-D .«» 

S? iS 

E^ 



S -E 
S|CN2 



^ X 



c 
c 



a> 

O in 

w c 

01 o « 



3 

iz >- 
re o 
J= 

r 01 

~ i 

o > 

W X 

*-/ * 

■£ u 
re 

j: uj 
u 

o « 
2 u 

re E 

^ *£ 
o ™ 

Jt E 
u >^ 

V o 
f * 

0) 

re ■> 



-D O 

o; 

™ in 

re o; 

OJ re 

re 3 "D 



O) 
■52 C. 
re E 
•Si re 

X 

— C 
I C 
re 
•£ 

^ OS 

in -» 

re .£ 
0) 

E £ 
» ^ 
X o 
> 
sz 



T3 .re 
C in T£ 

TOO)"" 

^ 1 s 

0 Q. 3 
>. X re 

01 w « 

E iS •£ 



in ~ 

m re 

i = 



3 

03 



VI 

< 



2 

o 
o 



O in w 



C °. 
O 

w 

"O re 

<U o 

^ i 

"D 
■D — 

3-6 

O 

-C O 
in 

in 

.r: a> 

II 

° E 



in q; 0) 
*' — in 
re 
S3 



0) in 
in re — • 
0) U O 



s_ O 

O E 

=i -E 

_q; c 

— a> 



0) 
■D 
'> 

O 

O 



$ :S ^ 



in 



in ■— 

iS J 

0) "D 

•£ 2 
I- o; 

0; Q. 

CL 0; 

a-° 

>- 5; 
~ re 



c 

<« 00 

c E 

a> o 

JD u 

in 

O W 

c 

in Q. 

DO u 



VI — 

2 »2 



2 c 
u a> 



w re 



3 
U 

I 

ro 

O 



O in 
Q. 
C 3 

q; o 

5 00 
a; in 

00 2 

re .0 

^ 

re ™ 

£ > 
a; 
> 

-— in 

C U 

= •= 

?l 

.2 00 



E ^ 

~ > 



E - 
.£ 1 

c ^ 

^ -D 

1 ^ 

> 

re ^ 

S C 

2 £ 



c 
.0 

re 
•D 

re 



<u re 



op:; 



01 O 



* !5 

u => 

.E 

.1 

1^ .y 



2 

Q. 

E . 

re to 

X u 

0) -- 

Q) O Z3 

in ^ I- 



^ 5 

U m 

s >- 

Q. O 



re i 

01 V c 



cr ™ - 

£ -D O 

^ :> ^ 

^ I 

O .£ > 

5 00 .c 

in C 00 



00 o 

-E P ■ 

O ^ 00 

U J3 E 



0) DO 

3 -i 

2 § 

re 

52 

0) V 

-c ^ 
u — 

re flj 

a> re 
00 ^ 
re 01 



0> 
■D 
re 



re 
3 

DO 
C 

'c 
00 



= X) .P 



c 



Q. 

re i- 

c -s 00 
re 0) 

S ^ •- 

Ol O 0' 

I-.E ^ 

t ^ re 

u ™ E 

< >2 >2 



5 -5 
2 5 

0) 



3 

.ii O 



DO 

c 



-P _^ 



o 

5 



m " 



i= DO 



*J 2 

.4— in 

o c 

2 5 



o ™ 
c o 

0) — 
C V 

o c 

Q. O 
O in 
Q. o 

<z Q 

a> in 

11 

^ re 

2 Q. 
re Q. 

J3 re 

-O y 
00 ^ 

.E re 

3 c 



*-* k- flj 

^ ^ 



(/» O 

Q. 5 

Si ^ 
00 

X -o 
_eu .2. 
Q. T 

11 
u a> 

" s 



O) 3 
CL O 
in _o 



01 g; 



re 

XI u 

2 "5 
c 2 

" -n 

|1 

E aj 



S > 
<y o 



re 

a> 

c 
re 

._ U 

c S o 
a. ^ J3 



0) 



01 M 



o •= 

I- c 



in 



c 
\j re 
.!2 *u 



c 0> 

<u c 

C 0) 

O DO 

E " 

8| 

I = 

q o 



^ 



Z 2 .5 
5^ 



c 

§! 

DO 
in 

o> .y 



0, j£ u i; 0) -s 

^ tn m — 



in 
0) "o 



s -a 



V o 

> ^ 

a> m 



a> re 

in r- 

o > 

Q. w 
in 3 

0.0 

re 

2 u 



u u 



re 



01 

3 



01 

-o 00 

o I ^ 

in a; 

w >-: 

re E ^ 

5 u o 



1 0) re 
E 00 

2 c 5 
« ™ re 
O 



is u 



re 

E 



%n A) 

o o 

00 E „ 

o o S 

£ o> 

re ~o in 

in C E 

V — >r 

o 2- « 



-D 

O 

-D 

o; 

cx 
re 

-D 
re 



O 
oi 
oi 

^ .y 

- E 

0) "D 

E ^ 



in 



o* -= 

in O 

in CL 

re 

2I 

re 01 

E ^ 



in 



w - .ii p S !^ 



O) ^ 
3 

W in 2 

m = O 

re a; w 

^ Q. 

ro . 

O O) 

00 U 

!S £ 

£ X 

Q. 

X ^ 

01 C 

o) E 



J3 
3 

O 

o 

o; 
> 
o 

0) 



2 i 

§2 
1 £ 

jO re 



in 

^ in 

.. 01 
m 

-o 



t; re .ii 



s > 

0 01 

^ >- 

01 qj 
3 



£ "S 



>- 0) 



-D 2 

§ ^ 
o o> 



O) ^ 

S 2 



-S.E 



c o 
3 c 



0) 

in O 

U O 



£ .y 2 



fS o 

0) — 



2 - 

o ^ 
Q. 5 
01 
-^T in 
C 



o; oT 2 
o K 



in P 

•y i 

3 5! 



in ri 

3 re 



E 2 IS 



S -p 



Q. .S^ 

in 

01 
> 
01 



P U 

8^ 



ERIC 



-I r"'0 



a 

Z3 



X 






X 










X 






X 


X 






X 
























X 


















X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 










X 






X 




X 






X 


X 


X 












X 


X 




X 


X 


X 


X 




X 


X 












X 


X 


X 








X 


X 




X 


X 






X 




X 






X 


X 


X 


X 






X 


X 


X 


X 


X 


X 




X 


X 


X 


X 






X 


X 


X 


X 




X 


X 


X 


X 


X 


X 






X 


X 


X 


X 


X 


X 


X 




X 


X 




X 




X 




X 


X 




X 


X 


X 


X 


X 


X 


X 


X 


x 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 



lOOJd "91 

wqa8|V -SI 

sdiqsuoneiSii pue sujanej 'C i 

S|eiU!39(3 pUB SUOllDEJJ 

A]i|jqBqoj(j puB ssusiij^^p 

9suas |B!JBds puB Ajjamoao '6 

uoijBjnduioo jaquinN 9|0qM *8 

suoijBjado JaquinN 3|0MM P sjdaDuoo 'i 

uoiiBJauinM puB asuag jaquinN *9 

uouBiupsg •£ 

suoijDauuoo iBDUBUiaqiBw *t- 

SujuosBaj] SB SDiiBUiaqiBW *E 

uopED!unujui03 SB SDUBUjaqiBVN 'Z 

SuiAjOg luaiqoJd 'L 



a. 

M 

o 

Si 
>- 

2 



0) 

E 
cs 
c 

km 

3 
O 



u 
U 



CD 



V 
00 

■o 



X 



o 

CO 



> 



c 
o 



01 

c 

v 
D 



3 



o 

3 

m 
>- 
c 
re 

O 

X 



c 
re 

E 



BO 






htnin 


ars 


wers 


op 


v 


o 




as 


t- 



0) 

E 

o 

00 

o 
X 



s>|SBi JU9UUSS9SSV apBJD-qynoj jO sadAjojOJd pue spjepuejs WiDN I '-"nSi 




2 




ical 




ned 










■s 


E 




"i/i 

V 


c 


(1) 












c 


re 

fc 








assess i 






b 




have 


z 






asks 




o 
*^ 

(/) 














pes 


\r. 


the 


eral 


toty 








15 


re 




m 




pre 


y 


r 


hes 


V 
tn 


ec 


re 


u 

3 


V 


a. 


u 


O 










"re 




E 




3 




(1) 






E 




E 




o 




're 










c 


re 






not 




u 




u 


ea 


> 


i_- 


WJ 


that 


we 


we 




o 


o 




o 


a 







re 
Q. 



00 

c 
'E 
o 



re c 
O. re 

1/1 01 



3 

.o 



.- o 



re 
O 

00 



(U re <" 

> 0) a< 

J -o 

o "re 3 

P •- O 



2r 
> 



£ E ^ 9 O 



c 

3 

E 
E 
o 
u 

DO 

o 

I 

E 

JD 
O 



■- D. 



JS w 

> 2 C = o 



3 
U 

■■5 

re 
Q. 



O 

z 



2 C 
J2 re 

-D ^ 

" re 
■!2 E 

3 a> 

O N 

w 

re 
Q. 

O P. 



11) m 
C ^ 
00 — 

-I 

J2 P 



0) u 



re 



£ Z 

.t: ^ 

S -P 

(U O 



00 

•- 00 

V DO 
DO 

■±5 IC O 



"S o 

IP c 

c re 

iS V 

«" E 



re a; 

E 

^! Si > 



re 

E 



i 2 

O i- 

u _re 

T3 3 

c u 
re 



'J 

(1) >- ™ 
.£ re 

S ■ 

-— .tr O «n 
- - 3 O 



_ _ 
Q, — 

^ w C 

re c 0) 

E •= 03 



<u .y 

^ re 

I a; 

j: ■£ 

1/1 re 

J E 

o ^ 



S " 

Q. 
^ (/» 

3 « 

00 E 
tZ o 



J3 0, 

wi f— **' 

« s « 

•£ c j= 

S c 
o g- h 

r> B ^ 

i; p 

re 

^ ?. .2 

L. O - 

fO DO w 



i! re 



ERIC 



CD 



n 
E 

V 

o 
o 
j= 
w 

w 

»2 
t/» 

n 
o 
O 




CD 



-c. 
p 



2 
o 



ERIC 



01 
01 

-o 
c 



re 

E 

01 

re 
E 

01 

0) 
w 
U 



c t: 
re re 

re eb 



01 



J3 

DO > 

re O 
E ^ 



o; 



re u. 

1 ^ 

C Q. 

OJ Q. 

u re 



E §f 
o J 



^ O) <2 



c 
o; 
> 

LU 



c 

01 

E 

0) 



c 

3 

Q. CO 
O 



o> o; 
> ' 
o; 



I 
DO g 

O) 3 

j:3 00 



*/) re 



DO 

c 



oj o 

" c I 

0) g 

g -5 o 



Q. — 

a 01 

■p: "O 01 .. 

00 £ 



o; 



o; 



3 

ro 

(J 
<U 
JO 

vO 

re 
c 

01 



o 

00 

o; 
> 
re 

O 
Q. 

O 

c 



3 

o 

U 

c 
c 
ro 



■£ ■£ r- 



ro 



o; 
O 

u 



o; 

01 



Si -c 



a; 



3 

01 0) 
01 



^ o 

^ u 

.5P o 

ro w 

>: ^ 

0) 0) 

O) "S 

ro E 

a; OJ 

oi 

c 

H- re 

5 u 



c 
re 

E 
u 

vD 



re 
o> 



c 
re 



-D O 



_ 01 
00 I'* 
3 



S 5 

re E 

o; ^ 

re rs| 



C 
ro 
o 

3 



0) 

a! 
E 

o> 

Q. 



O 
O 



re 

E 



c 

3 

o 
E 

ro 
ro 



ro 



T3 
01 



a. 
E 
o 
u 
o; 
J3 

c 
ro 
u 



DO S 

IG 

C 01 
3 J= 

o 

^ CO 

^ c 
01 5 

m t 
ro <« 

ni o; 
2: ui 

O ro 

1/1 VJ 

re 

re E 
o; o; 
> -c 

Si re 

^ E 

Q. .£ 

f § 
II 



O 

w <" 

2 ™ 

w 00 

■c E 

QJ Q. 

— 

C H- 
— U 
■o Z 

01 
00 

re w 

DO c 
C 01 

o) E 

^ V, 
^ o. 



3 -D 
U 01 
m 

0) re 
^ J3 



o; 



o; 



3 0) 

y ■§ O 
'd. ro E 



O 
I 

CL) 



q3 u 

= 3 



o; 



VI 



3 
O 



T3 
01 
> 



ro 

E 
o; 

JO 

re 
E 



•5 
> 
c 



t re 
o; 

Q. E 

E ^ 

O 3 

o "E 

3 

t/i tj 
"D ^_ 

re O 
-D 
C 

re "O 

tn (5 

II 



0 o 

1 ™ 

2 c 
re ^ 
Q. E 



3 

— re 

. 0^ 

CO E 

LU 

I/) W) 



12 iS "E 2 •§ ^ 



O X 
O) 



u 
ro 



O) 

u 
c 
re 
> 

re 

o ro 

:5 E 

-o .£ 

" 0) 

— ro 

E 

c «> 

0) •= 

C QJ 

S -a 

in ^ 

re ■£ 
01 

E 



Em 



E -B Oi 3 



— > u 
•c <^ 

3 T3 
w O) 
ro m 
0) ro 

l/l 

<« 45 5. 

O) ^ 

Q. c X 
« >• IS S 
o; o -c 

2 p o o 



.b; 3 



^ _, 00 -D 
- ° c c 



0) 

c 
>■ 

o 

o 



O) O «;-" 

^ 00 ^ 

^ro « 

^ -2 ^ 

> 0) 5 

c J= > 

o o 



ro 12 
01 00 o £ 

3 ^ r~ 

>n 0) Q. .r 
1/1 f" **— 

w J3 *- 



OJ 

re 

OJ 

-o 

T3 
C 
ro 

T3 
o; 

tn 
3 
(J 



o; 
3 

o; 



Q. 
E 



3 

y c 

£ .2 u re 00 



m — _ 

0) > c 

00 ^ — 
ro 

tn C r- 

ro t 

E B ° 

in Qi 

C w 

O — 

£ § 

ro 



01 



01 



T3 (5 

c ^ 
ro O) 

<^ u 

QJ .= 
QJ m 

i 

>~ 01 

> ro 
o, E 

.E 
o Z 
c 

ui QJ 

q3 E 

v> QJ 
C 



OJ 
Q. 
>- 

o 
o 



a 

<^ re 
i/i 



T3 C 



O O 

c E 

3 X 



|.y CO 

00 <f 

3 

o c 



QJ 



c 
a; 

E 

in 
01 



5 



^ D. re 
.§ o 



ERIC 



SJxeh8dfin«r«h«ch»d(«filouni«m«ot ThtlgoftbelowthwwIheftsulUoltitOMnMpliytdMlaf. 




{Rtmtnibtr, In tha piclurt. an arrow lika 



Riaant lhal Jota won hi* gam* agalnal Alax. Tha arrow always point* ttom lO ^ '°*«r.) 



i 



1 . Who woo tha gama batwaan Pat and Robin? 

2. Which chiWran has Laa already played against? 

3. WWchol those game* dkJ Lee win? 



4. How many gemesheve been played by Ihe 
chiWren *o far? 



Explain how you know. 



5 Make a table showing the current standings ol the six ehiWren. Put the player who has the most games in first 
place, at Ihe top. II two player* are tied. Ihey can be feted In eilhef order. 



Name 



Win* 



Loses 



1. 
2. 
3. 
4. 
5. 
6. 



S. The tournament will be over when everybody ha* played everybody else exactly once. How many more games 

need to be played to finish the toorrwment? . 

Explain your answer. 



7. Dana and Lee have not played yeU Who do you think win win when they play? 
Explain why you think so. 



7i 



James has rented a rowboat to row in the pond around the playground. 
On the playground there are three pieces ol equipment: 



a play fort: 



an umbrella: 



and a merry-go-round: 



n T 



Look et Ihe mep. Pnd Jsfrs* In his boel. Imegln* how the pleygroond took* 
to James. From where James was. Ihe playground looked like this: 



5E±9 

The merry-go round was on Ihe lell. Ihe umbrella was on Ihe right, and the 
fort was in Ihe middle. 

1. James rowed In Ihe pond lor a while. When he Ipoked at the playground a second time, this Is what II looked Ska: 

Figure oul where James was when he saw the playground Ihls way. Draw a dot there and label it A. 

2. Then James rowed some more. He came to anolher spot, and slopped. When he kxjked at the playground Irom 
this spot, it looked dillerenl. From here, Ihe playground k>oked like this: 






Wiiere was James now? Please draw anolher dot on the map lo show where James was end label It B. 

3. James really enjoyed rowing, so he rowed some more and explored some more. Alter a while he was at the spot 
marked C. 

Please draw lha lort, Ihe umbrella, and Ihe merry-go round In the spece below, to show Ihe wey the playground tooked 
to James when he kjokcd at it Irom Ihe spot maiked C. 

4. II was a warm, pleasant day, end Jemes ihought thai maybe he woukJ keep on rowing eH eflemoon. He rowed to a 
new spot on the pond. When he looked toward Ihe playground, he we* *uddenly *urpri**d. He could no tonger *ee 
Ihe umbrellal AH he couM *ee we* the Ion on the lelt. and the merry-go-round en the right. 

Pier .c draw a dot on the map to *how where Jame* va* when he diecoverad that he could not **e the umbrella. 
Label it D. 

5. Do you think ihet J.tme* coukJ row to e spot on the pond where he see* the fori on hi* left end Ihe umbrele on hi* 
rigW. but he cent *ee the merry-go-round? Plee** explelo how yoo know. n(j 
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BY AucE J. Gill and Lovely H. Billups 

Tt was 2-30 on the afternoon of February U, 1990. In 
ianinncr<it%' classroom in Rochester. New York. Chap- 
ter 1 Basic Skills teacher Marc>' MiUcr stepped mto the 
secondicnde classroom prepared to condua her twice- 
weekly math lesson. She approached the homeroom 

•Mrs. Jones. 1 know you re having a Valentine s 
party this afternoon. Tell me how much time I have for 
a math lesson.* ^ ^ . ^^. xa^ 

-Miss MiUer.* came the frustrated reply, "these children 
were so disorderh- this morning. 1 don't think they re 
poinp to have a piny. You can have the rest of the after- 
noon." A young girt immediatel)- raised her hand. 

•Does that mean well have math tonger today? 

With convioion in her voice that bespoke a punish- 
ment well delivered. Mrs. Jones almost beamed as she 
affirmed. -Thafs rightf whereupon, the class began to 
cheer and applaud. . , _ 

Seven-year-olds happily forsaking a Valenune s Day 
party for the opportunity to do more math is not a pic- 
ture that typifies our classrooms. In fan. report after 
report casuAmertc«nyoungsters-andadults-as being 

uncomfonabie wtth mathematics, indeed, often express- 
ing an intense dislike lor the subjea.' Math, they feel, is 
best left tt) engineers, scientists, and a small elite gn>up 
endowed at birth with a ulent for the arcane world of 
numbers. 

Alice J. cm is an assistant director of the educational 
issues department of the American Federation of Teach- 
ers. As a third-grade teacher in Qeveland. Ohio, she was 
o te of the developers of Thinking Math. Lovely H. 
BiUups is director of field services of the AFTs educa- 
Honal issues department and coordinator oftu Educa- 
tional Kestarcb and Dtstemination program. 
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It's not surprising that large numbers of Americans 
have an aversion to math. Most people <teUke activiues 

they're not goodat. and on that score the figunsare 
ning. The inadequate achievement of U.S. students m 
mathematics has been chronicled in headlines, papers, 
books, and conferences. In a 1991 study conduaed by 
the EducationalTesting Service. American nine-year-olds 
ranked twelfth out of fourteen industrialized nations m 
math, ahead of only Stovenia and Portugal; American thir- 
teen-year-okis tanked sixteenth out of twenty. On the 
1988 International Association for the Evaluation of Edu- 
cational Achievonent. for ten-year-<rtds. 38 percent of 
American schools scored below the /ou«« scoring 
school in Japan. On the 1991 National Assessment of 
Educational Progress, fewer than half of the twelfth 
graders demonstrated a consistent grasp of decunals, 
peicents. fractions, and simple algebn. 

Doing something about the American dilemma m 
math can at times led like trying to move a mountam; It 
has been there a tongtimc. Yet, though still rare, the real- 

life scene described in the opening of this article is 
becoming more common as efforts proceed to transform 

America's dassrooros into places where children not 
only k)ok forward to math but master it well enough to 
successfully compete on a gtobal levd. 

WE GO NEXT to a classroom in Lake County. Florida, 
where students are ti Inking about a class picnic. 
They know that iwenty^bur students and seven adults 
wiU be present. One of their tasks is to dedde how many 
six-packs of soda to buy so each person can have a can 

of soda. In small table groups, they move counters re|> 
resenting the cans and finally determine that they need 

to purduse half a doien six-packs. The problem isn t 
cxtnoidinary. tt's t two«ep pfobkm that requires stu. 
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aenu to nuke « substantive decision about how the 
(MjainderinadiwisioaproblefflsbouklaffeadKanswec 
•n^ if one type of probtem with which American $cvj 

^fKMde students have had difficulty on National 
]^]i^^nem of Educational Progfcss (NAEP) tests'. What 

is unusual in this instance is that the problem was sue- 
cessfullj sohred by these Lake County first gnders. 

The lesson was a demonstration of Thinking Mathe- 
matics, a researdvbased approach to teaching mathe- 
outics that grew out of a collaboration between the 
American Federation of Teachers (AFD and the Learning 
Research and Development Center (LRDQ of the Uni- 
versitv of Pittsbur]^. The philosophy of Thinking Madv 
ematics, which is consistent with the standards adopted 
by the National Council of Teachers of Mathematics, is 
that we can produce not only students who are capable 
mathematicians but also a populace that appreciates the 
place of mathematics in their own lives and that is tJO 
longer "mathophobic* 

In Thinking Math, value is placed on thinkmg. rea- 
soning, communicating mathematically, focusing on 
relationships, usmg what is known to find the unknown. 
Its content and sequence is shaped by the idea that, right 
ftom the start of their mathematical education, children 
can and should be engaged in the discussion, analysis, 
and solution of mathematical problems: they need not 
wait unta. step by step, they have mastered a strict hier- 
archy of basic skills. It draws heavily from the cognitive 
apprenticeship model, which emphasizes the impor- 
unce of working on authentic, real-life tasks and of 
exposing students to the reasotung and strategies that 
experts employ when they acquire knowledge or put it 
to work to solve such tasks. 

This approach is in sharp contrast to the milieu out of 
which teachers have come, which emphasized memo- 
rization and rote procedures. The majority of elementary 
teachers have scant background in mathematics. 
Because it has not been expected of them, few have 
taken amthing beyond high school math, except for a 
college course in elementary math methods. The com- 
mon image of the proper way to teach math, held by both 
teachers and the public, is based on their own memo- 
rization and formula-driven school experiences. On the 
elementary' level, the core of aaivity in math classes cen- 
ters on what are called 'the basics.* This consists of mem- 
orizing basic facts and rules and perfbmung page after 
page of computation. Probably 90 percent of the math 
aatvities in elementary classrooms have been tradition- 
all}' conduaed to fMdlime this memorization and calcu- 
lation. 

In addition, studies indicate that math instruaion in 
the United States is repetitious and pooriy organized. 
Teachers spend weeks and sometimes months each year 
repeating: ihe content of previous years. In many 
in.sianccs ihc students appear to have 'forgoiicn.' when 
in faa. they may never have really learned the concepts. 

We now know with certainty that this traditional 
approach to math education will not produce thinkers, 
interpreters, and users of information. 

WHEN CHILDREN come to school, excited and curi- 
ous, they already have ways of thinking about 
quantities and numbers. What usually happens is that 
they encounter a teacher who begins to model and, with 



Right J^om the start of their 
matbenuaical education, children 
can and sbouSd be engaged ir, the 
discussion, dnafysis, and solution 
of mathematical problems; they 
need not watt utOO, step by step 
they have mastered a strict 
hierarchy of basic skills. 




the best of intentions, demand that the children mirror 
'the correct way' to think about and manipulate num- 
bers. If this 'correa way* is the child's way, the child 
becomes a star performer, if not. the child becomes con- 
hised. loses confidence in himself, and begiris to form a 
disastrous opinion about his ability to do math This is 
not to say that everything a child thinks is true or valid. 
Yet children play counting games, share, and make pur- 
chases at the store before they come to school The% 
ihink about numerosity and measurement in the worid 
around them, choosing the biggest piece of pic. the bag 
with the most cookies, arguing about who s taller than 
whom. Using the p er sp ective and knowledge that chil- 
dren bring to school is the first of Ten Principles that con- 
sutute the Thinking Mathenutics approach to teachmg 
mathematics. Although the phrase 'start from where the 
children are* has echoed thiough the hatls of schools ^ 
decades, it has referred to where they are in rciauc 
our curriculum and not to how diUdren think cr I 
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1. Build Ctoni incuidve ioiowledte. 

2. EmbUsh a straof number sense throu^ counting 
esticpation, use of beacfanmks, mental compuution 
skiiis, and imdentancaag iiic cuccu ui opcnuions. 

3. Base instruction on situational story problems. 

4. Use manipulativesand other representations to rep- 
resent the problem situation; then link concrete an J syro- 
b(riic representatiODS. 

5. Require studems to describe and justify their math- 
ematical thinking. 

6. Accept multiple correct solutions and, in some 
cases, more than one correa answer. 

7. Use a variety of teaching stntegies. 

8. Balance conceptual and procedural knowledge. 

9. Use ongoing, new assessments to guide instruction. 

10. Adjust the curriculum timeline. 

In addition. Thinking Math (TM) reconunends that 
teachers focus on depth instead of quantity', that math 
classes be used to look at a few problems from many 
angles rather than to work many problems the same way. 
This belief, which grew out of the research findings, is 
similar to the praaices of Japanese and Chinese class- 
rooms where the goal is lasting conceptual understand- 
ing and where it is not untypical to devote an entire class 
period to one or two problems." 

The picnic soda scenario described earlier was one 
pan of the exploration that day. In addition to anending 
to children's intuitions, the lesson visibty incorporated 
two other principles of TM. Ftrst. the students had phys- 
ical objects (manipulatives) with which they could 
model and. thereby, strategize about, a £uniliar situation. 
Thus, numbers, quantities, and operations had meaning 
beyond paper Secondl>-, the students were grappling 
with a problem that had meaning to them. In ThirUcing 
Math, teachers are urged to write or change problems to 
reflect their own classroom, school, or city and to incor- 
porate aaivities about which their students express 
interest. This not onh' increases motivation but also 
shows that math is. indeed, conneaed to the real world. 

As a classroom is opened to students' thinking, there 
begin to emer}!e multiple intuitive and inventive ways of 
solving problems. The encouragement and support of 
this process is a crucial pan of the teaching and learning 
in TM classrooms. Again, there is a strong parallel to real- 
life situations. Woe to any commercial enterprise that 
sees only one wax- to solve a problem! If that one solu- 
tion becomes stymied, the business may fold unless an 
inventive mind finds another way to come at the prob- 
lem. Both studenu and teachers come to appreciate this 
principle. Nine-yearckl Bruidon, addressing the Ander- 
son. Indiana, school corporation one night, observed. 
"In Thinking Math, there arr a tot of wa>'sof doint* a prob- 
lem so we can ch(K>Mr the best way* 

In Albuquerque. Neu- Mexico, another ninc-vcaM)ld 
who had transferred to the public schools from a private 
one the year Thinking Mathematics was introduced, 
shared how he felt about the program. "My other school 
thought they were real good, hut this one is better. Thc\ 
only uught us one way to do a problem, but we icamcd 
a lot of ways here.* And still another student s voice: 
•What do I think about Thinking Math? It helps vou think 
a tot better Not only letting kids do it different ways but 
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tattinit kids do it their ofwn wi\' . . . you can think with 
your head and do matih the wav'vou need to do it to solve 
problems and sQifflike thaL' 

1^" 'KsSc uJrrSSSnT soategies ue dormant and unvaii- 
dated, however; ynlessdieyareshared within a com- 
mtmity of learners. Tbm, reqt^ring students to explain 
and jtistify their strat^es is another important Tltitiking 
Mathematics principle. This puUic discussion about 
mathematics (mathematical discourse) adds power to 
students and teachers in three ways. It requires the speak- 
er to clarify his own thinking. It allows other students to 
get additional p er sp ec ti ves on a problem. It gives the 
teacher infbrmatton about the level of understanding a 
student has. information that is not evident when a teach- 
er tooks only at compuution. 

A second-grade dass in Lake Counry, Florida, had 
worked through a problem that centered on a typical ele- 
mentary school VUentine's Day occurrence. Children 
bring in cookies and theyare shared by the dass. One of 
the goals of the lesson was to demonstrate how number 
sense can be tised to stmplify the scrfution in a way fre- 
quently used by aduhs but rarely uught in school . When 
adults buy rwo items that cost $ 1 .98 each, they often cal- 
culate two times two dollars and subtraa ifour cents 
rather than go through the regrouping algorithm. The 
problem the second*grade students were solving 
required them at one point to add 36 and 29, the num- 
bers of cookies brou^t in by two students. A stuffed 
replica of Curious George (whom the teacher often used 
as a friendly voice to introduce other ideas into the dis- 
cussion in a nontntimidating way) suggested that if you 
could add 30 instead of 29, it woukl be a realh- siniple 
calculation, which he could do in his head. 
•36 -f 30 is 66." 

•Could we do that?" asked the teacher. Not two sec 
onds passed before young Jessica raised her hand and 
said, •Yes. But you have to take one away from that 
because you added one too nuny. So you'd have 65 cook- 
ies.^ 

The role of the teacher in these exchanges is extreme 
ly imponant. Encouraging students to do their own 
thinking and to find many ways to solve problems is rtot 
the same as telUng students to "go discover* while the 
teacher sunds back and watches. The teacher becomes 
planner extraordinaire, fiamer of the circumstances that 
will enable students to find their way Generally, the solu- 
tion strategies are those that are brought forward by stu- 
dents as they think and use the materials provided. In this 
instance, the teacher intetjeaed "another way* to think 
about the problem. For the process to avoid turmng the 
classroom into a venue where the teacher is seen as cu>- 
todun of -the right way," the teacher must know whicti 
concepts or strategies studenu will probabh' not find 
intuitively and makea judgment about when and ho« lu 
ininKJiice those ideas. 

Finally, the teacher must be capable of moving m direc- 
tions suggested by student conversation and of focuttng 
thai conversation on mathematical kleas, targeted and 
untargeted. To do this successfully, the teacher must 
develop a good sense of the mathematical territory and 
of students' conceptual understanding. The studem » 
explanation must be dear enough for other students tc 
foUow the thought. In instances where it is not cVnr 
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where the teacher believes some students need a coo- 
aete modd. the teacher appropriately ni$e$ ?«» 
or asks the solver to concretely demonstrate wnai nas 

bcssssk! 

Leadii^ mathnr***^* discussioas is a complex task. 
Students long accustomed to traditional math dasso 
oiay be leluctant to e^qwess their ideas at first. Thc/iuat 
ity of discussioo iiiqprw« ow time as students becon^ 
aware that their Ideas are vahied, as they learn to express 
themselves, and as teachers, too. leam from actual con- 
versations. Teachers must beco.ne knowledgeable about 
what actually demonstrates understandmg. To return 
again to the 36+29 Vdentinc cookies problem, an expla- 
nation that basicaUy does nothing more than repeat a for- 
mula (e.g.. 6+9»15. put down the 5. carr>- the 1; 
1 +3+2»6: 65) does not show the same evidence of num- 
ber sense that investing quantities with their proper 



Brandon's Ten Good Reasons 
To Like Thinking Mathemajics 

HI! I'M Bnmdon Sokol. I come from the family 
class at Robinson School. I am going to give ten 
reasons why I like.Thinking Math better than tradi- 
tional math. 

1. Thinking Math eliminates the need to memorize 
a problem cause we don't learn to use paper, 
we leam to use our heads. 

2. In Thinking Math we can make up and solve 
our own sichawashunol story problem. 

3 . We are also able to use manipulatives to help 
solve the problem. In Thinking Math, a manipu- 
lative is a small block or tool to help count with. 
This (displa>'ing a base ten rod) is ten. This is a 
hxmdred. This is one. And this is a thousand. 

4. We can also use decomposition to break down 
a problem. Decomposition is breaking down a 
number, for instance. 378 would be 300+70+8. 
That would be easier to add to another number. 

5. In Thinking Math there arc a lot of ways of 
doing a problem so you can choose the Best 
wa>'. 

6. In Thinking Math we leam to do our math in 
our head so when we go shopping we can add 
numbers fuL 

7. We discusaa problem to make sure everyone 
knows the steps to the problem to complete 
the problem so that the next aren't so hard. 

8. In Thinking Math we use our own record book 
of our knowledge not someone elsc '!> 

9. We only need to do 3 or 4 problems dunnt; 
math. 

ID. With Thinking Math a group of children can all 
do one problem together. 

<— EUtANDON SOKOL 
AgfV 

AMrtmitf thtAttStrum. ImSimiui. 
Sdtool Corpitrmuim Bnanl >y Truattt 



roeaningdoet<30+20is50: 6+9is 15: 50*15 is65>. 

One young man in Cleveland, near the end of the first 
year (^Ttainkiiig Math, turned in the foUowing as a solu- 
tion to a sidxiactioa problem. 

541-268 

5O0-20O«3OO 

41 -68 ••27 

300-27 « 273 

This youngster had not been »ught this method. But 
he had been in an atmosphere that clearly valued explor- 
ing number territory and finding different ways. He 
would undoubtedly have been a good -tradiUonal- math 
student He became, for his shon time in Thinking Math- 
ematics, a marvelous inventor who saw how things fit 
together and what was happening with the quantities in 
this opeiatioa. He knew 41 nunus 68 was mmus 27. he 
said, because be took away 41 and then he still needed 
"fifiy-one. sixtyooe Cbe bad raised two fingers) and 7 
mote for 68. So that's twenty iboiding up tbe two fin- 
gers, ead> <tfwbid> stood for 10) seven.- This student s 
explanation encompassed a way of thinking about 41 
minus 68 that his teacher wouW not have thought of 

WITH ALL the promise of the current mathematics 
scene, theie sdU exists a good deal of skepticism. 
Parents worry about whether their diildren will leam 
basic £icts and be able to compute, the centerpieces of 
their own math education. Those whose chiklren are 
now grown worry about repeating the disastrous path of 
'new math' from tbe sixties. It is, ther ef o r e, important 
to clarify the relation of Thinking Mathematics to both 
of these concerns. 

First, the concern about learning basic math com pu-^ 
ution skills. One of the Ten Principles of TM ts to P(|fl^ 
vide a balance between conceptual and proceduff||p 
learning. This means that, where formerly It was accept- 
able if students were able to rotely compute even if thc>- 
didn't understand what they were doing, this is no kmgcr 
considered exemplary performance. Even the 'iiood* 
students who went through -drill-or-kill" tactic* ire 
unable to compete with their peers worldwide * One at 
the deficiencies of these students is their inabiliiy to rea- 
son and solve complex problems. 

Teachmg for conceptual understanding takes kMiper 
than rote learning, k does, however, make it po*Mb»e for 
students to also acquire basic knowledge without an 
overemphasis on uiunotivatmg, unconnected dnB and 
the calculation of bedsheet-length sets of problca» TM 
deemphasizes the amount of time spent on teaming 
basic facts, believing that students will leam them « ihry 
work with them. Walk down the halls of a schooi where 
teachers are domg what they have always done and rou 
will find a belief that "until students know thoM basics* 
It IS ii waste of time to tum their attention to «omnhinK 
a^ complicated as word problems. "Thought pcotoinm' 
at the ends of textbook chapters are genenlly tkipprd 
because "there simply isn't time available if the midema 
are to master their fictt." Visit aTM class and you wrfi fM 
students engaged m sohring problems about fMwIiar m- 
uations: some know the bcu, others are stil tcammg 
them, but all can successfully solve interesting protiAr!!^^ 
You may find some periods of short (5 to 7 mmMc* > oj^p 



on teu. If you do, the drill will often have a 
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Tbe children are so excUeO! 
They're always asking to do math, 
tve never had ihat happen before. 




schema behind it. 

7+8 
5+6 
8+9 

In tiiis ocercise, students ieam thinking strategy for 
'near doubles.' Doubles are learned fiurly quickly. If stu- 
dents know the doubles (e.g., 7+7) they can projea an 
answer that is one more than the double. Or, the drill may 
promote a mental math skill that will be useful in com- 
puution. 

17+10 
?.8+10 
52+10 
69^10 

There is evidence that students who spend less time 
on drill and practice do not suffer when it comes to 'the 
basics.' In foct. the O^tively Guided Instruction pro- 
jea, an approach similar to Thinking Math, found that 
students in classrooms with a problem-solving focus 
aaualty outscored their peers. Additionally, a study by 
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SKI Intanatioaal ibund: 

By comparison with conventional practices, 
ittstniakxi that «^rh««*^ meaning and under- 

jriHtng <J S8CSC CftCilVC SI iuCUSCSmSS MVauiCCu 

lUSItthtUltastiif^ffKtitmaiteaebingbasicskms, 
and cngifes children flxxe extensivdy in academ- 
ic learning (anphatis added).* 

Even the results of standardiaed tests— which are not 
geared to measure the kinds of conceptual understand- 
ings Thinking Math aims for— put TM students above 
their noo-TM coumcipatts. Researchers from the Uni- 
vetsity of Pittsbuigfa compiled data from student scores 
on standardized tests fior the group of TM pUot classes in 
1990^1. Hie report^ of this dau sated: 

Standardized achievement test scores indicate 
that pro^ students did at least as well, if not bet- 
tei; than dieir noo^pcofect peers on both Compu- 
tatioo iOdCoocqics and ^>plicatioiu subsections. 

Class scores were averaged for all students in grades 
one tfaroui^ five for TM classes and non-TM classes for 
each of the subsectioiu of standardized achievement 
tests administered in die kxal districts;* the results are 
presented below. 





IMcbMc* 


Non-TM classes 


Computatioo 


64 


59 


Concepts and 
flpplifationt 


66 


50 



In addition, the report continued: 

Notable improvements in student problem-solv- 
ing abilities were indicated by results of the prob- 
lem>solving test.* The percentage of correct 
answers on the post>test exceeded that to be 
expeaed from the item difficulty established by the 
test makers. 

. . . (T)here are multiple indications that student 
learning and attitudes were entianced by their par- 
ticipation in the Thinking Mathenutics program. 

The spontaneous observations that Thinking Math 
teachers have made not only support the dau regarding 
student achievement gains but also point to qualiutive 
differences as wdL One teacher in Brevard Cotmty, Flori- 
da, observed, Tbey are so excited! They're always ask- 
ing to do math. I've never had that happen before.' 
Another commented, "I'm now teadung my chiklren 
how to think and understand math, not |ust how to do 
math."* Still another from a pikx class observed, 'My sto- 
dents are being exposed to and coining up with skills I 
would not have thought possible.' Another reflected that 
the students were showing greater appreciation for the 
use math wouM have in their lives. 

THE SECOND concern about a new approach to 
teadung mathematics is the lingering aun of distaste 
from the 'new math' of the sixties. There are several 
ways in which Thinking Math diffen. One is that 'new 
math' was imposed upon teadicrs by people perceived 
to be out of toudi with daisrooos. A resuh of the actu- 
al top-down stmegjr of this momnent it that it did not 
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provide, in tiine or in siteaiice. what teacfaen nec^ 
to undentand die proccM and the madi diqr were beti^ 
asted to teach. Because tcidien stood at die core of oe 
j-.-»j-^jMj£ QfThinkinc Mathenutici. thcv were abic 
rakkm^diese needs. Thinkios Math alao has provid- 
ed that fnunework of research that profaukxuls can use 
to inform what they ou(^ to do and why. While sproc 
concepts that weie pan of 'new math' aurfue again, 
ther suitee wtdi a latkMiale and a profeaaiooal devel- 
opment effort diat aBowt teadiers to make meaning of 
them. Anodierpiimcpioblem with 'new math-was that 

patents were left out of die equation. 

In discmsing the resistance encountered by 'new 
math,' theNationalReseardiCouncilnoies, 'When 
parents could not or did not understand the need 
for dunge or the reasons new curricular emphases 

(of diat program) were chosen, resentment and 
anger resulted, and a conviction set in duu if the 
•old math* was good enoui^i for them, it was good 

enough for dieir chiWrcn.' Farenu have become 
enthusiastic about programs when they have been 
given the opportunity to ask questions and have 
them answered, are involved in their children's 
learning, and are assured that their cfaikiren's com- 
putational skills and learning outcomes are not suf- 
fering." 

Thinking Mathematics advises its teachers how impor- 
tant it is to communicate widi parents as diey begin the 
prognun. and. on the whole, diey have had a very posh 
tive response. ,. 

Not onl>' has the developmental process for Thmkmg 
Math been different from the one for "new math.* there 
are also substantive differences. As it was implemented 
in classrooms, "new math* concepts were taught 
abstractly. Since this is not how chiklren best learn math, 
the goal of developing a deeper understanding was not 
achieved. For example, 'exfnnded noution* (e.g.. writ- 
ing 485 as 400+80+5) was often used in 'new math." But 
the instruaion was generally based on abstraa. con- 
tcxtiess figures. 

In Thinking Mathematics also, students frequently 
recofd solutions using expanded noution. However, this 
nows from a construction of the meaning of those quan- 
tities that starts when students manipulate and group 
objects; compose, decompose, and recompose numbers 
many ways or they solve problems about bmiliar things 
and in ways they understand befan they are introduced 
to efficient algoridwis. So their understandings of bodi 
quantities and operuions are richer and more lasting and 
diey can successMhr attack unftuniliar situations. 

The 'new matfr* also lacked a sutrounding system of 
student dialogue about mathematics that helps children 
develop and clarify their thinking while allowinp the 
teacher to better see the depth of their understanding 
and where they falter. Nor was it grounded in a philoso- 
phy that accepted and promoted multiple ways of solv- 
ing problems. 



CHANGE IS never easy The history of education 
reform is dunercd with great ideas that never took 
root in the classroom. Coundess manuals, materials, and 
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nanipulatives. once imbedded in such hope, are situng 

somewhere cotteoing dust. 
TOnkiotMadietnatics is one program dut has not suf- 

fefcd sudi a tee. It is Aacdoaing in duity<ight dttes. 
wtth a wittttM list of donns nicie. A 1992 ssvr/ cf p=r 
; teadiers found tfiat they were making signifi- 
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nnt f^ng f* in drir msthrni***^ ttwrniction. On aver- 
agc. diey were using strategies lecommcnded by Thinlji 
ingMadidiree dmesa week, or 60 percent of die ume.^' 
To understand why Ihinking Madi app^ to be tak- 
ing hold where other reforms failed, we have to go back 
a few years to tlie start of die American Federadon of 
Teadiers' gducafiow«» Research and Disseminadon pro- 
gram. EX&D began in 1961 with the goal of enhancing 
teachers' professional development by giving them 
access to an rrr rrr ^^g knowledge base of classroom 
research and involving them with the educational 
reseaicfaen doing that researdi. As AFT president Albert 
Shankernotedatdietime. teadiers will be denied recog- 

nidoo as p tofcs sfona ls undl diey can demonstrate that 
their actioiis and judgments are grounded in a solid pro- 
fesskmal knovHedge base acquired through intensive 

and continuing study. 
In the past, much educational research was packaged 

in ways that were remote from teachers' daily experi- 
ences. Most studies were abstxact and }aigon laden, so 
it's not s m pi isiog that le acher s tended to find them of lit- 
tie use, if they read them at all. When mandated from 
above, as many 'researdhbased* teaching programs 
were, they were met widi suspicion from teachers and 
produced superficial change, at besL For dieir pan, few 
researchers seemed to know or care about teachers' per- 
cepdons and were unable to relate to the realities of the 
classroom. 

AFTs ER&D program sought to reverse those negativ^^ 
trends, to encourage teachers to value the informatioi^jp 
available from authentic research findings, and to 
expand dieir tools of practice. Originally ftinded by die 
National Institute of Educatkxi, ER&D is a long-range, 
peer-to-peer, unio^sponsored strategy for professional 
improvement that encourages teachers to become users 
of research. 

A KEY COMPONENT of the program is the develop- 
ment of 'research translations' that highlight and 
interpret the most important researdi findings in practi- 
cal ways that teachers can use; translations have been 
done on a wide anay of topics, from coopeiativc small- 
group teaching to student mothration. The translations, 
based on a singk; work or a group of related works, elim- 
inate ponderous sttdstics, interpret technical language, 
and focus on the practical applications of the research. 
AS part of die process of devekiping ER&D materials, 
experienced teachers woric coltaboratively and inten- 
sively with researchers from universities and education- 
al research laboiatories. 

But the translations are only the first step in the pro- 
cess of putting valuable rrsearch findings in teachers' 
hands . Through die ER&D networic teachers in local AFT 
affiliates receive training in the basic tnuislations and in 
peerteaching techniques. Once trained, these teachers 
in turn esttblish focal programs and tnin other teachm. 
spreading the program's beneftu to a growing numb^^ 
of dassrooo teachers. Tbe focal tiaineis meet periotVlF 
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fh€ bottom HiteqftbitemUavor is 
mot books or numttais, but tbe 

tSeepet 9p*esUon of j UtutiSy tiuermg 
what goes on hi tbe classroom 




cally with AFT naUonil staff to review nuterial. share 
their experiences, and learn about new research trans- 
lations. A three-pan philosophy guides the ER&D train- 
ing: Sessions are non^hreatening. non-judpmental. and 
voiuntar>'- 

Thinking Math provided the ER&D program's Tirsi 
move into specific sub)ea maner. previous translations 
had deah with more generic teaching skills. Suned in 
1967 through a National Science Foundation grant to the 
AFT and the Learning and Research Development Cen- 
ter at the University of Pittsburgh, Thinking Math built 
on the ER&D tradition of dose collaboration between 
teachers and researchers. Lauren Resnick. LRDC's co- 
director and one of the first reaearchcrs to imena with 
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WUS^vtathm, dwc ri bci the <3q;>eneiic e this w«»-: 

Tt/t VUiD pcopin enables teachers to probe. 
qHeadoaaDdiacenawitfaeducational researchers 
aboKtheirfiDdiflgL bisunkiue because it does not 
create a d Ufcieuce in laak or pcestige between 
researchers and teachers. It stresses that 
researchers and teachers need each other— that 
educational effectiveness is not the exclusive 
pfovince of eitber gro^>. 

Combining the dinical wisdom of teadiers and the 
rich research background of the cognitive scientists, the 
coUabontioa has to date produced two Tlmtlnnf A/atd 
volumes, coveting counting, estimating, adding, sub- 
tracting, multiplying, and dividing. The (oint effort has 
also resulted in die pidiiicatioaofa recent book, Anafy- 
sis ofArtAmgUc far MaOMmatics Ttaebing (Lawrence 
Eilbaum), that discusses current research knowledge rel- 
evant to teaching math in grades one through eight. 

THE BOTTOM line of this endeavoi; however, is not 
books or manuals, but the deeper question of actu- 
ally akerfng what goes on in the dassroom. And here the 
evidence is cause for optimism. For example, as the sur- 
vey mentioned above found, only 19 percent of the 
teichers said they bad encouraged their students to solve 
pttiblems in more than one way before becoming 
invcritvcd with die program; after their TM training. 71 
percem were focusing on multiple strategies. 

Tbe effectiveness of the training depends in large pan 
on the degree of cooperation between the local affiliate 
and the school district Tbe districts where the program 
appears to be flourishing have provided teachers rdease 
time not only for initial training, but also for regtilar fol- 
k>w-up sessions. 

One of those districts is Anderson, Indtana. where the 
various partners have shared the cost of building a solid 
Thinking Math program over the past few years. The AFT 
has paid for training two local Thinking Math coordina- 
tors, and the distria has picked up the tab for traming 
seventeen other kical teachers (including the cost of sub- 
stitutes and release time once a month to allow teachers 
to reflea together on their experiences). The Anderson 
Federation of Teachers has also contributed significant 
amounu of money in the past three years to expand the 
program. By the end of this school year, every elemen- 
tary teacher in Anderson will have had an introduaory 
session on Thinking Math. 

The experience of one Anoerson teacher of learning- 
disabled studentt provides an insight into the power of 
the Thinking Math approach. The teacher convuiced 
three of her colleagues that all thdr students ctHiM learn 
together if they ttt^t math the TM way. With the teach- 
ers working as a team, the program has proved so suc- 
cessful that it is difficult for an outsider to distinguish the 
leammg-disabled students firom their classmates during 
.nath lessons. 

In Albuquerque, New Mexico, teachers at one school 
have seen the power of Thinking Math in helping (hem 
break their professional isolation. The teachers amnged 
their master schedules to provide periods of tune for 
teachers of the same grade to collaborate and talk about 
their math lessons. In the phikMOFrfiy of die Japanese 
(ContinuMl OH page 48) 
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I^IINKING Maihemaucs 
(Ckmtinutdfrompt^ 11) 

teachers who caieftilljr hone each les- 
son to peneviiuu, iiiey uiii tiiui iheir 
"polishing time* 

Rhode Island provides another 
promising model. There, the Rhode 
Island Federation of Teachers has 
brought a Thinking Math team to the 
state to woik with teachers from sev- 
enl districts. As the local teachers 
wrestle with their own initial imple* 
menution of the progiam, they 
be able to meet monthly throughout 
the year, which will provide a level of 
muttial kuppon for the teacher lead- 
ers that had not been possible for 
other teams. 

npHESE STORIES illusttate the kind 
X of commitment that Thinking 
Math requires. The program asks 
teachers to lethink their most basic 
beliefs and assumptions about teach- 
ing and learning mathematics. Such 
radical change cannot be brought 
about by one-shot 'professional 
development* workshops or by plop- 
ping manuals into teachers' laps. 
There is no siibstitute for the coUegial 
and research-based process that per- 
meates the ER&O training: it provides 
a forum and suppon network for solv- 
ing the problems that arise when 
teachers make substantive changes. 
To get past the ineviuble bumps in 
the road that accompany change, 
there also needs to be a non-threat- 
ening atmosphere, sufficient training 
that continues after the initial train- 
ing is done, and opponunities for reg- 
ular interaction with colleagues. 

Some teachers who are trained in 
Thinking Math have been troubled 
when they do not return home with 
a set of discrete and sequenced activ- 
ities. But they come to realize that the 
program requires that they rccon- 
strua their teaching, using their local 
curriculum, from their new knowl- 
edge and beliefs. More than 90 per- 
cent have been able not only to make 
this adjustment for themselves but 
also CO successfully inspire, tram, and 
pass on their abiltrv' to their peers. 

R<.fonnmg the way mathematics is 
taught would be accomplishment 
enough for the program. But its 
ejects are more sweepmg. Thinking 
Math has convinced those intimately 
involved with it that the best route to 
genuine education reform is through 
a new look at content. When teach- 



ers passionatet>* believe that new 
approaches are necessar>' and pro- 
ductive in their dail>' teaching (of 
math, in this case), they begin to see 
tnat Changes fflusi be matic m tne 
eiftire structure of schools to accom- 
modate and suppon those new 
approaches. They begin to rethink 
how the school day should be orga- 
nized: they come head to head with 
standard assessment practices and 
realize they need overhauling: they 
redesign staff development and con- 
sider new ways of organizing school 
staff. Whatever stands in their way 
gets dose scrutiny, and rvhat starts as 
Thinking Math often adds up to much 
more. □ 
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CONNECT 

Bringing meaning to matii 
with a student-run store 



wholesale 



retail 



loan 



inventory 
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by Deborah Black 

IMAGINE THE range of skills and concepts 
that can be addressed in the undertaking of a stu- 
dent-run math store. Skills include estimating 
simple sums and differences, multiplying with 
money, introducing decimals and many others. 
A 5% markup of wholesale items or a 10% dis- 
count on sale items offers a wonderful lesson on 
percentage! 

Fractions enter the picture when you con- 
sider cuning a pie in eighths and selling 
individual pieces. Consider the complexity of a 
word problem that begins with the price of the 
ingredients in oatmeal cookies. How many cook- 
ies are in a batch? How much will you sell the 
individual cookies for if you want to account for 
a 5% profit? Keep in mind all the measuring in- 
volved in making the cookies and the 
importance of following the recipe directions. 
For some students, learning to make change for 
a dollar is a formidable task in itself. 



Goals 

The academic goals of our store were clear at 
the onset. The store would provide students 
with an opportunity to develop math skills with 
an improved attitude towards math. Students 
were continually engaged in problem solving at 
every level of store operation. Collegiality and 
cooperative group work began to unfold. 



Fresh ideas 

Each year the store has a fresh beginning as a 
new group of students bring to it their ideas and 
student-made merchandise. The only dimen- 
sions of the store that remain constant are the 
enthusiasm, excitement and ongoing learning. 
Students take charge of their own learning and 
begin to make connections between panicipat- 
ing in different aspects of running the store and 
understanding certain math concepts and skills. 

Preparing for the math store was the math 
curriculum for the initial weeks of school. Con- 
cepts and skills were reviewed or introduced as 
they related to some task in running the store. 
Peer teaching and apprenticing practices were 
developed in the first steps of the preparation 
process so that ongoing problem solving related 



to the store could take place in an atmosphere 
of support and collegiality. Later in the year stu- 
dents ran the store independent of other projects 
going on during math class. 

Establishing respect and support 
A lot of thought and consideration went into un- 
derstanding what it meant to work together with 
a common purpose. Tune was spent defining a 
mission for the store and a code of ethics for all 
participants. It was cleady stated in their code of 
ethics that all students had the right to participate 
in every aspea of running the business. It was 
made clear that it was the responsibility of the 
group to make sure everyone fek supported and 
treated with respect as they learned new skills nec- 
essary to run their business. Peer teaching and 
apprenticing were concepts that had been ad- 
dressed earlier in the class. Once the foundation 
for working together was established, the process 
of preparing smdents to mn their store began. 



Defining jobs 

A division of labor was established and a job de- 
scription for each job was wrinen by the 
students. Job titles included salesperson, cash- 
ier, buyer, inventory clerk, record keeper and 
even crafts people and bakers. Later on the stu- 
dents added to their list of jobs: loan officer, 
advertisement coordinator, and entertainment 
coordinator. 

Each job description, as defined by the stu- 
dents, also stated what types of skills were 
necessary to be successful at each job. For ex- 
ample: 

A salesperson needs to: 

• have knowledge of merchandise in stock; 

• be friendly and helpful to people, even when 
rushed; 

• be able to add simple numbers in your head; 

• know how to add the cost of merchandise to 
help the customer determine if he/she has 
enough money to pay for the item(s); 

• know how to discount sale items. 
These descriptions were always subject to 

change as students discovered new areas of ex- 
penise they felt were important to each job. 
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Practice run 

Before the students actually put their store into 
operation, we had a practice week in math class. 
Each student brought in an item for under a dollar 
to sell at the "trial run." We went through every 
step of running a store that they would encounter 
ordering merchandise, markup from wholesale to 
retail prices, opening and closing procedures for 
setting up the till, selling, making change, record 
keeping, keeping an inventory and re-ordering. 
Centers were set up so that every student was able 
to participate in all the aspects of iiinning the store 
during practice week. 



What to sell 

Careful consideration should be given to this 
matter. We decided to sell school supplies, nutri- 
tional snacks and student-made crafts. We also 
decided to sell items that were under a dollar 
and put a dollar cap for spending. We wanted 
everyone to have an opportunity to participate 
in the shopping and did not want money to be- 
come a status issue. 

Start-up costs & physical set-up 
Students wrote a letter to the PTO asking for a 
loan of S65.00 to invest in stock with a payback 
schedule attached. 

Once the loan came through students started 
preparing for their grand opening. Each student 
signed up for their first job. Those who were 
hesitant about their ability to perform certain 
tasks could be an apprentice to a peer teacher. 

For a physical set-up, nothing elaborate is 
necessary. We used a roll-away cart that we actu- 
ally rolled into different classrooms. Designing 
a math store structure would be a wonderful 
construction project! 



The grand opening 
The opening date was well advertised in the 
5th/6th grades. With a bustle of energy and en- 
thusiasm, the grand opening was a success. 
Over time, changes and additions to store rou- 
tine became apparent and needed to be worked 
on collectively. Students took on more responsi- 
bility and began to organize baking groups and 
encourage student artists to sell their wares. 
They were motivated and challenged by the de- 
mands of their customers and their drive to 
make their store a success. 

Regular follow-up and reflection 
.After every store closing, students would work to- 
gether to prepare an update report for the group. 



At a group gathering smdsnts presented a finan- 
cial report and an update on inventory items. A 
discussion followed that allowed smdents to re- 
flect on the store — ■ 

operations and make 
suggestions for im- 
provements and 
changes. Task forces 
were established to ad- 
dress issues brought 

up in the discussion 

and each student chose a job for the next open- 
ing of the store. 

Evaluation and assessment 

Noting the math skills and concepts covered by 
running different aspects of the store made it com- 
fortable to assess student learning by watching 
them prepare and perform their job. Students also 
kept a math journal to reflect on the trials and 
tribulations of mnning the store. This served as a 
valuable tool for assessing and evaluating smdents 
learning. Attending to smdents' participation in 
problem solving related to the store was also valu- 
able in assessing their development and 
understanding of math concepts and skills. 



Students were continually 
engaged in problem solving 
at every level of store 
operation. 



Extension and integration 

Many opportunities for integration present them- 
selves as students discover new ways to serve 
their customers and operate their store. For in- 
stance, smdents decided to sell only "healthy" 
snacks at our store which became a springboard 
for discussing nutrition. One year we had an- 
other class make jewelry and we sold their 
merchandise on commission in our store. 

The store has been a wonderful oppormnity 
for smdents to actively engage in hands-on 
math in a meaningful way. Each year as a new 
group of smdents comes to the class ready to be- 
gin their own business, the extent of 
involvement and complexity of the operation is 
determined by the collective needs of the sm- 
dents. The same process and strucmre for 
preparing to operate the store and the actual run- 
ning of the store provide a skeleton from which 
I operate. However, the details and meat of the 
matter are dependent on the smdents and the ex- 
tent to which they are dev^lopmentally ready 
for the math concepts and skills covered in the 
different aspects of running the store. ^ 

DEBORAH BLACK has a BS. in Communications 
Disorders and a M.Ed, in Deaf Education and has taught 
for 10 years overseas and in the United States. 
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Employer expectations for school 
mathematics 



Experience with problem 
formulation is exactly what 
we need in business and 
industry. 



by Henry O.Pollak 

WHY DOES SOCIETY give us so much time 
to teach mathematics? There are. of course, rea- 

sons internal to the 

educational system: 
This is needed to pass 
the test, that is needed 
for the next course, 
etc. At this point, I am 
more interested in rea- 

sons outside of 

education itself. We need to teach mathematics 

• for practical everyday life; 

• for intelligenrcitizenship; 

• for employment; 

• as pan of overall human culture. 
There is much overlap among these pur- 
poses. We will focus our attention on the 
mathematics needed for employment. 

My views are based on 35 years of experi- 
ence in the telephone industry, most of it in the 
Bell System prior to divestiture. As a mathema- 
tician. I had the job of sticking my nose into 
everybody's business. On the basis of this life- 
long experience of myself and my many 
colleagues, here are some thoughts about mathe- 
matical expectations of employers: 

1. Employees need to 
know how to set up a 
problem, that is. to 
take a situation in the 
real world of work 
and formulate a pre- 
cise question the 
answer to which 
would help with the 
situation at hand. 

2. They need to be 
able to figure out 
what an answer 
should look like be- 
fore they Stan to 
solve. If the problem 
is numerical, for ex- 
ample, about how big 
should the answer be. 
and what son of preci- 
sion should they aim 
for? 



Expectations in the Workplace 
for School Mathematics 

✓ The ability to set up problems with the ap- 
propriate operations. 

✓ Knowledge of a variety of techniques to 
approach and work on problems. 

✓ Understanding of the underlying mathe- 
matical features of a problem. 

✓ The ability to work with others on prob- 
lems. 

✓ The ability to see the applicability of 
mathematical ideas to common and com- 
plex problems. 

✓ Preparation for open problem situations 
since most real problems are not well for- 
mulated. 

✓ Belief in the utility and value of mathe- 
matics. 

(PoUak. 19S7) 



3. Once the problem and a desired accuracy are 
defined, the employee should know, or know 
where to find. SOME method of solving it. 
What method is used is not imponant! 

4. The employee should know that it is possible 
to look at a tremendous variety of practical situ- 
ations in an analytical, structured, systematic, 
quantitative, i.e.. "mathematical" way. 

5. Employees need to be able to work together, 
to function as a team, to help each other 

6. They need to be able to communicate, to get 
across by the spoken and the written word what 
they are thinking about and what they have 
done. 

These are simplified, abbreviated statements 
of need, but I think that business will recognize 
their imponance. 

Let me now switch hats — but still on the 
same head — and ask what these needs say to 
the processes and content of mathematics educa- 
tion. How is all this related to the Standards, 
and to the many current movements and 
changes? Here are some funher thoughts: 

1. A key difference between "NCTM's Agenda 
for Action," from 1980, and ihc Standards al- 
most 10 years later, is the evolution from the 
prime emphasis on problem solving in 1980 to 
problem finding, before you try to solve, in 
1990. Experience with problem formulation is 
exactly what we need i--' business and industry. 

2. One effect on mathematics education of the 
rapidly changing technology is the even greater 
need than before for teaching estimation (at the 
developmentally appropriate level). This is also 
exactly what the employer needs. Before you 
Stan any problem, you ought to have an idea 
how big the answer ought to be. You also need 
to know, as was pointed out above, what preci- 
sion is required. When you have done both of 
these, then you can consider alternative meth- 
ods. If it is a numerical problem, do you want to 
do it in your head, or on a calculator, or by pa- 
per and pencil, or on a computer? It is terribly 
imponant that future employees have learned to 
do all four of these and have developed some 
(metamathematical) judgement about when to 
use which. 

3. There is a great variety of mathematical 
thought. In the past, the bulk of students — i.e. 
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future employees — have at best seen three of 
these in school: geometric, arithmetic, and alge- 
braic. In the world of work, we also meet 
situations which are inherently uncertain. The 
information is essentially probabilistic, and stu- 
dents must learn how to deal with it. There are 
other situations which are data-driven, and fu- 
ture employees need to know how to look at 
data and draw potential conclusions from them. 
Many real-world situations require planning, op- 
timizing, choosing among alternatives. Thus the 
beginnings of the mathematics we call operat- 
ing research ire very useful. Many real-world 
situations aie combinatorial or graph-theoretical 
or algorithnic in lature. and students should 
therefore have seen these kinds of mathematics. 

4. The fact that it is possible to look at so much 
of the worid in mathematical ways is the es- 
sence of mathematical modelling. Students need 
to have experienced this. 

5. One of the most helpful developments in edu- 
cation in recent years is cooperative learning. 
When student.": learned only the behavior of 
competing with one another, one of the first, 
and most difficult, behaviors which employers 
have had to teach is cooperation. Group work 
will, we hope, help with this problem. 

6. Language across the curriculum is another 
very hopeful and valuable trend. A piece of 
work is of no use to the company if it is neatly 
engraved in the employee's head or notebook, 
but unavailable to anyone else. Knowing how to 
speak and how to write is essential in making 
yourself useful to your employer. Writing in 
mathematics will help to prepare the future em- 
ployee for this: so will oral repons of work. 

The competitive position of the United States 
in the twenty-first century makes it essential 
that we do better in mathematics education than 
we have in the past. An awareness of the needs 
of employers will help to move the System in di- 
rections which will benefit everyone. 

HENRY 0. POLLAK is a Visiting Professor at Teachers 
Coiie^elCoiumbia Unn ersiry. and retired Assistant Vice 
President of Bell Communications Research. Inc. *Si 
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Evaluating Problem Solving 
in Mathematics 



Effective assessment of problem solving in math 
requires more than a look at the answers students 
give. Teachers need to analyze their processes and 
get students to communicate their thinking. 

Walter Szetela and Cynthia Nicol 
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In its Curriculum and Evaluation 
Standards for School Mathematics, 
the National Council of Teachers 
of Mathematics expanded the goals it 
developed in 1980 for promoting 
problem solving as a curricular focus 
(NCTM 1989). The first three stan- 
dards — Mathematics as Problem 
Solving, Mathematics as Reasoning, 
and Mathematics as Communication 
— show a shift from, emphasis on 
rules and routine problem solving 
dominated by teacher talk and passive 
learning, to active smdent participa- 
tion, in which reasoning and commu- 
nicating are stressed. 

These efforts are admirable, but 
they create new challenges, especially 
in assessment of these higher-level 
skiUs. Problem solving requires 
considerable thinking, but even when 
smdvints are able, they are not inclined 
to communicate their thinking. 
Without such communication, how 
can we reliably assess students' efforts 
to solve problems? Before discussing 
how to improve communication and 
assessment, it is useful to clarify the 
notion of a problem and problem 
solving. 

The Nature of Problems and 
Problem Solving 

Problem solving is the process of 
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conftonting a novel simation, formu- 
lating connections between given 
facts, identifying the goal, and 
exploring possible strategies for 
reaching the goal. A problem, then, is 
a simation in which the individual 
initially does not know any algorithm 
or procedure that will guarantee solu- 
tion of the problem, but the individual 
desires to solve it 

Success in problem solving depends 
upon metacognitive processes, as 
described by Garofalo and Lester 
(1985). The following list summa- 
rizes the typical sequence of actions 
for successful problem solving: 

1 . Obtain an appropriate representa- 
tion of the problem situation. 

2. Consider potentially appropriate 
strategies. 

3. Select and implement a 
promising solution strategy. 

4. Monitor the implementation with 
respect to problem conditions and 
goals. 

5. Obtain and communicate the 
desired goals. 

6. Evaluate the adequacy and 
reasonableness of the solution. 

7. If the solution is judged faulty or 
inadequate, refine the problem repre- 
sentation and proceed with a new 
strategy or search for procedural or 
concepmai errors. 

182 



These metacognitive processes are 
difficult to assess, but assessment can 
be expedited by creating problem situ- 
ations that facilitate students' commu- 
nication of their thinking. 

Difficulties in Assessment 

of Problem>Solving Performance 

The difficulty of assessing complex 
processes necessary for solving prob- 
lems is exacerbated by the failure of 
smdents to communicate clearly what 
they have done or what they are 
thinking. Smdents are prone to make 
calculations without explanations, and 
calculations alone often fail to reveal 
sufficiently the nature of the solver's 
work and thinking. It is not enough to 
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ANALYTIC SCALE FOR 
PROBLEM SOLVING 



Understanding the problem 

0 • No attempt 

1 • Completely misinterprets the problem 

2 - Misinterprets major part of the problem 

3 - Misinterprets minor part of the problem 

4 • Complete understanding ot the prablam 

Solving the problem 

0 • No attempt 

1 • Totally inappropriate plan 

2 • Partially correct procedure but with 

major fault 

3 • Substantially correct procedure with 

minor omission or procedural error 

4 • A plan that couki leed to a correct 

solution with no arithmetic errors 

Anawering the problem 

0 • No answer or wrong answer based 

upon an inappropnate plan 

1 - Copying error; computational error; 

partial answer tor problam with multiple 
answers; no answer statement; answer 
labeled incorrectly 

2 • Correct solution 
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check for right and wrong answers or 
to use multiple-choice formats for 
assessment of problem solving. As 
Silver and Kilpatrick (1988) sute: 

A reliance solely on the sleek effi- 
ciency of muldple-cboice (and other 
short answer) formats will severely 
hinder efforts to help students 
develop a reflective and interroga- 
tory stance toward their learning. 

If we can devise methods for eliciting 
better communication of students' 
thinking, we can perform more effec- 
tive assessment. Such assessment 
measures the quality of students' 
thinking. This information can help 
teachers design and implement 
instruction to promote greater success 
in problem solving and can help 
administrators evaluate programs and 
curriculums. 

Assessment of Solved Problems 

The most natural and common method 
for assessing performance in problem 
solving is to obtain general impres- 
sions about the quality of a solution 
while scanning students' work. These 
general impressions are strongly influ- 
enced by the "proximity of correct- 



ness" of the answer. As a result, good 
solutions with minor errors due to 
carelessness that alter the answer 
dramatically can receive undeservedly 
low scores. Scales are available that 
focus more attention on solution 
procedures, enabling teachers to 
obtain fairer and more reliable scores. 
For example, Charles, Lester, and 
O'Daffer (1987) devised a scale that 
assigns separate scores to each of three 
stages in problem solving: under- 
standing the problem, solving the 
problem, and answering the question. 
Figure 1 shows a modification of their 
scale, with increased emphasis given 
to the understanding and solving 
stages (Wilson 1991). 

The Charles, Lester, and O' Darter 
scale and its modified forms are easy 
to use. An advantage of such a scale 
is that a teacher may focus on only one 
of the stages. For example, a teacher 
who is emphasizing strategy selection 
and implementation can assess each 
student's solving procedure irrespec- 
tive of the answer. 

The California Assessment Program 
(Pandey 1990) includes comprehensive 
descriptions of various levels of perfor- 
mance for specific problems. This is 



CATEGORIES OF RESPONSES IN SOLUTIONS TO PROBLEMS 



Answer 

1. Blank 

2. Undetermined 

3. Incorrect 

4. Correct 



Statement 

1 . No statement 

2. No context 

3. No units 

4. None reauired 

5. Complete 



Strategy Selected 

1. Number sentence 

2. Select ODerations 
and calculate 

3. Algebraic 

4. Non-systematic list 

5. Systematic list 

6. Guess and test 

7. Draw diagram 

8. Look for pattern 

9. Logical reasoning 

10. Use simoiercase 

11. Work backwards 

12. Undetemiined 



Implementation 

1. No work shown 

2. Identifies data only 

3. Problem misinterpreted 

4. Strategy not clear 

5. Strategy initiated 
(table, graph, list) but 
incomplete or poorly 
implemented 

6. Conditions or possibilities 
overlooked 

7. Multiple secondary errors 

8. A single secondary error 

9. Appropriate and complete 
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Problem solving 
requires considerable 
thinking, but even 
when students are able, 
they are not inclined 
to conununicate 
their thinking. 



appropriate for large-scale assessment 
programs. However, the classroom 
teacher has litde time to construct 
■cales for individual problems. 
Teachers need assessment procedure^^B 
and scales that they can modify or us^^ 
intact for a wide range of problems. 

Categorizing Responses 
to Problems 

Scales for assessment of problem 
solving can be designed without 
creadng an evaluative threat to 
students. Such a system of scales was 
constructed for use in the 1990 British 
Columbia assessment of problem 
solving (Szeteia 1991). Instead of 
scoring the solutions only, teachers 
analyze the responses to problems on 
the basis of four categories: answers, 
answer sutements, strategy selection, 
and strategy implementation (see fig. 
2). Teachers can use a single category 
to determine how well their students 
are addressing a particular aspect of 
solving problems. One focus might be 
on strategies used. Another might be 
directed toward answer statements. 
Incomplete statements that fail to 
include the units taught or importani 
contextual information may serve as 
focal points 'or teachers in their subse 
quent instructional activities. 
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Students are prone 
to make calculations 
without explanations, 
and calculations alone 
often fail to reveal 
sufficiently the 
nature of the 
solver's work 
and thinking. 



Promoting Greater Communication 

To further enhance assessment, we 
need to devise problem situations and 
questions that encourage and motivate 
students to communicate ind explain 
their thinking. Figure 3 shows one 
way to do this. An already solved 
problem with a significant error, 
combined with a set of relevant ques- 
tions about the solution, facilitates 
communication. As witli an unsolved 
problem, students must form a suitable 
representation of the problem. Instead 
of solving the problem themselves, 
however, they analyze the given solu- 
tion. Finally, they reveal their 
thinking by answering the pertinent 
questions. Answers to these questions 
can provide more comprehensive 
insights about the student's thinking in 
problem situations than more typical 
problem formats, in which students 
may have various levels of success but 
fail to reveal their thinking. 

Assessment of responses to the 
questions accompanying the already 
solved problem can be done in less 
time than it normally takes for 
teachers to plod through the usual 
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wide range of solution procedures for 
a given problem. The main goal of the 
example in Figure 3 is to determine 
whether or not a student understands a 
problem situation well enough to 
recognize the incongruity of the given 
answer despite excellent implementa- 
tion of a good plan, with the problem 
solver nmning awiy only in the care- 
less writing of the answer statement. 
Teachers can provide continuing expe- 
riences for students to critically 
analyze solutions and conununicate 
their observations and responses to 
relevant questions. Such 
practice can help students 
engage in reasoning, eval- 
uating, and communi- 
cating, and can enable 
teachers to assess these 
problem-solving 
processes more 
effectively. 

Other forms of prob- 
lems with questions to 
stimulate thinking and 
written communication 
include the following: 

• Present a problem 
with all the facts and 
conditions, but have the 
students write an appro- 
priate question, solve the 
completed problem, and 
write their perceptions 
about the adequacy of the 
solution. 

• Present a problem and 
a partial solution. Have 
students complete the 
solution. 

• Present a problem 
with facts unrelated to the 
question. Have students 
comment about the 
quality of the problem or 
revise the problem to 
remove the incongruity. 

• Have students explain 



how they would solve a problem using 
only words, then solve the problem 
and construct a similar problem. 

• After students solve a problem, 
have them write a new problem with a 
different context but preserving the 
original problem structure. 

• Present a problem without 
numerals. Have students supply 
appropriate numerals, estimate 
answers, and solve the problem. 

Teachers can assess die quality of 
each response by using a scale such as 
the following: 



Example of Problem that Asks Students to 
Communicate Thinking 



A bowl contains 10 pieces of fruit (apples and oranges). 
Apples cost S cents each and oranges cost 10 cents each. 
All together the fmit is worth 70 cents. 
We want to And how many apples are in the bowl. 
Kelly tried to solve the problem this way. 

/O X 5 = 50 gx. 5-=H0 

2 K to =20^ 3 ^ /O:: 20 

70 «^ X (0 = 'fO 

Try to follow Kelly's wort( and solution. Then answer the 
questioru. 

1 . Is Kelly's way of solving the problem a good one? 

Tell why you thinit It is or Is not a good way. 

b«c*iit« 1+ '^i ll t^JI y«« He fo sibU 
j;/«l-i<»« wKicK if *-h*t l*"^ 
Shft Alln-f- pe»i tarfoiiy 

2. Old Kelly get the right answer? 

/Vo 

Explain why she did or did not. 
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1. No response or simplistic or irrel- 
evant response. 

2. A relevant response but of minor 
importance with respect to the ques- 
tion or problem. 

3. A reflective and significant 
response but with an important omis- 
sion or misconception. 

4. A comprehensive, logical, and cor- 
rect response to the question or problem. 

Tliese suggestions for assessment of 
problem solving have the potential to 
reveal much more than we currendy 
know about smdents' thinking, their 
concepuons. their weaknesses, and 
their strengths. With better awareness 
about students' knowledge and 
thinking, teachers can plan more effec- 
uve instruction, and the outcome is 



more likely to be better learning of 
higher-order skills essential to success 
in problem solving. □ 
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